Big Data/BI Zone is brought to you in partnership with:

Jakub is a Java EE developer since 2005 and occasionally a project manager, working currently with Iterate AS. He's highly interested in developer productivity (and tools like Maven and AOP/AspectJ), web frameworks, Java portals, testing and performance and works a lot with IBM technologies. A native to Czech Republic, he lives now in Oslo, Norway. Jakub is a DZone MVB and is not an employee of DZone and has posted 149 posts at DZone. You can read more from them at their website. View Full User Profile

Bash Magic: List Hive Table Sizes in GB

01.10.2013
| 3953 views |
  • submit to reddit

To list the sizes of Hive tables in Hadoop in GBs:

sudo -u hdfs hadoop fs -du /user/hive/warehouse/ | awk '/^[0-9]+/ { print int($1/(1024**3)) " [GB]\t" $2 }'

Result:

448 [GB] hdfs://aewb-analytics-staging-name.example.com:8020/user/hive/warehouse/mybigtable
8 [GB]	hdfs://aewb-analytics-staging-name.example.com:8020/user/hive/warehouse/anotherone
0 [GB]	hdfs://aewb-analytics-staging-name.example.com:8020/user/hive/warehouse/tinyone
Published at DZone with permission of Jakub Holý, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)