hadoop - How do I store data in multiple, partitioned files on HDFS using Pig -


i've got pig job analyzes large number of log files , generates relationship between group of attributes , bag of ids have attributes. i'd store relationship on hdfs, i'd in way friendly other hive/pig/mapreduce jobs operate on data, or subsets of data without having ingest full output of pig job, significant amount of data.

for example, if schema of relationship like:

relation: {group: (attr1: long,attr2: chararray,attr3: chararray),ids: {(id: chararray)}}

i'd able partition data, storing in file structure looks like:

/results/attr1/attr2/attr3/file(s)

where attrx values in path values group, , file(s) contain ids. allow me subset data subsequent analysis without duplicating data.

is such thing possible, custom storefunc? there different approach should taking accomplish goal?

i'm pretty new pig, or general suggestions approach appreciated.

thanks in advance.

multistore wasn't perfect fit trying do, proved example of how write custom storefunc writes multiple, partitioned output files. downloaded pig source code , created own storage function parsed group tuple, using each of items build hdfs path, , parsed bag of ids, writing 1 id per line result file.


Comments

Post a Comment

Popular posts from this blog

javascript - RequestAnimationFrame not working when exiting fullscreen switching space on Safari -

request animation frame stops working when exiting fullscreen clicking on original space's safari window. fine if fullscreen mode canceled escape key or calling cancelfullscreen(). steps reproduce: open https://dl.dropboxusercontent.com/u/769042/prezi/safari-fullscreen.html click "draw raf", kittie appears click "fullscreen", go fullscreen click "draw raf", kittie appears go space original safari was, showing "click exit fullscreen mode", click anywhere, out fullscreen click "draw raf", nothing happens what handling click calling: window.requestanimationframe(draw); which draws on canvas context: function draw() { ctx.drawimage(img, math.random()*500|0, math.random()*400|0, 100, 100); } i checked .hidden , .visibilitystate, updated correctly. tested on osx 10.9.3, safari 7.0.4 (9537.76.4). has workaround/solution other switching old settimeout? this sounds webkit bug 88940 : using reque
Read more

Python ctypes access violation with const pointer arguments -

i have api i'm trying wrap in python (2.7.6 on win7) code using ctypes. here's api: client_dllfunc bool clientapi search_exporttoclipcopy(clienthsearch handle, int channel, lpctstr filename, const time_t& from, const time_t& to, const bool* cameras, int length, bool usepassword, lpctstr password, bool includetextin = false, bool excludeplayer = false); the issue i'm having cameras argument; , i've had issue other const pointer arguments well. here's how i'm wrapping api: def search_exporttoclipcopy(self, hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, password, includetextin=true, excludeplayer=false): exporttoclipcopy = self.sdkdll.search_exporttoclipcopy exporttoclipcopy.argtypes = [c_int, c_int, c_wchar_p, c_long, c_long, pointer(c_bool), c_int, c_bool, c_wchar_p, c_bool, c_bool] exporttoclipcopy.restype = c_bool exporttoclipcopy(hsearch, csearch, filename, tfrom, tto, cameras, length, usepassword, pass
Read more