The map union value you currently have can certainly carry another map type
within. Here's how you'd probably want to define it:
{
"name": "metadata",
"type": {
"type": "map",
"values": [
"null",
"int",
"float",
"string",
The avro-tools jar is usually a standalone one, and if you do have a
standalone variant running it with 'hadoop jar' may cause classpath
pollution as hadoop also includes a (likely different) version of avro into
the runtime classpath.
Run it instead this way:
export
Could you also link to the articles that claim Avro containers are not
splittable? It'd be good to correct them to avoid this confusion.
On Thu, Jun 25, 2015 at 11:25 AM Ankur Jain ankur.j...@yash.com wrote:
Hello,
I am reading various forms and docs, somewhere it is mentioned that avro
]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(PostMapper.class);
job.setReducerClass(PostReducer.class);
Regards.
Anand
--
Harsh J
-with-Avro-Tools-tp4030458.html
Sent from the Avro - Users mailing list archive at Nabble.com.
--
Harsh J
the reflect APIs, but because Schema doesn't
have an empty constructor, I get a NoSuchMethodException when trying to
deserialize on the client.
-Joey
--
Harsh J
the method Schema.createMap(Schema myType), but the key is
Utf8 not String.
--
Harsh J
(Schema myType) method generates a MapUtf8, MyType, not
MapString, MyType.
2014-05-19 18:16 GMT+08:00 Harsh J ha...@cloudera.com:
You can pass -string to the avro-tools compile program, to make the
generated classes use String/CharSequence and not Utf8.
On Mon, May 19, 2014 at 1:37 PM, Fengyun
. Only alphanumeric characters and underscores are allowed.
Could someone shed some light on why is this restriction?
thanks, Yael
--
Harsh J
! for some of the files, can this function help me
in that?
Thanks
Amit
--
Harsh J
are coming, and is there a way to
correct the file now without loosing any data?
File was created using 1.7.4 C library
Thanks
Amit
--
Harsh J
You can do this, sure. You just need a schema of string type or something
similar.
Are you not concerned about the read time of the data you plan to store as
strings? Typically you write once and read more than once during processing.
Storing the data types in proper serialized form would help
to fix this issue?
One more doubt: Why we don't have AvroMultipleInputs just like
AvroMultipleOutputs? Any reason?
Thanks Regards,
B Anil Kumar.
--
Harsh J
(ReflectDatumWriter.java:143)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at
org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
Thanks Regards,
B Anil Kumar.
--
Harsh J
I'd like to not have there.
Essentially I'm looking for something like NullWritable where the key will
just be ignored in the final output.
Thank you for any assistance or guidance you can provide!
Best Regards,
Ed
--
Harsh J
. If this is a not a Pair-schema then you
must explicitly set the job's map output schema using setMapOutputSchema
Thank you!
Best Regards,
Ed
On Thu, Jan 16, 2014 at 6:47 PM, Harsh J ha...@cloudera.com wrote:
Hello Ed,
The AvroReducer per
http://avro.apache.org/docs/1.7.4/api/java/org/apache/avro
}) for nullable
fields? Because of this issue, we have to handle UNION types in an
appropriate way both normal values and null values as exceptions.
Instead of UNION type, why don't we use a 'nullable' property for any field?
--
Harsh J
technology bases... google-groups
REALLY makes it easy to collaborate and search for issues that someone may
already have solved. (certainly http://search-hadoop.com/?q=fc_project=Avro
helps, if only you could post from there as well, w/out having to jump to
another interface etc.)
--
Harsh J
Hello,
Did you mean to send this to the Kafka lists instead of the Avro one?
On Fri, Aug 30, 2013 at 4:08 AM, Mark static.void@gmail.com wrote:
Can you view Kafka metrics via JConsole? I've tried connecting to port
with no such luck?
--
Harsh J
to compress the avro files that i am writing, for that i am using
the latest Avro C, with deflate option, but i am not able to see any
difference in the file size.
Is there any special type to data that this works on, or is there any more
setting that needs to be done for this to work.
--
Harsh J
touching the
remainder of the scary datum? I would call it a 'conditional read' in that I
only want to fully read the datum if the datum meets some criteria.
Anna
--
Harsh J
) throws IOException {
while (values.hasNext()) {
output.collect(new Text(key.datum()), new Text(values.next().datum()));
}
}
}
public static void main(String[] args) throws Exception {
ToolRunner.run(new MyAvroJob(), args);
}
-Anna
--
Harsh J
. How does the second/third/...
splits get the schema (the schema is always written to the first split) ?
Thanks,
Lior
--
Harsh J
Yes, we seek to 0 and we read the header then seek back to the split offset.
On Aug 1, 2013 11:16 PM, Lior Schachter lior...@gmail.com wrote:
Hi Harsh,
So for each split you first read the header of the file directly from HDFS
?
Thanks,
Lior
On Thu, Aug 1, 2013 at 7:36 PM, Harsh J ha
code I've found refer to the 1.0 version.
Thanks,
Jerrell
--
Harsh J
)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
--
Harsh J
Oops, moving for sure this time :)
On Wed, May 1, 2013 at 10:35 AM, Harsh J ha...@cloudera.com wrote:
Moving the question to Apache Avro's user@ lists. Please use the right
lists for the most relevant answers.
Avro is a different serialization technique that intends to replace
the Writable
for avro. Like
the total sort partitioner , which I think currently assumes writable as the
io mechanism.
I faced problem using with avro , so though of writing to the forum.
Thanks a lot
Rahul!
On Wed, May 1, 2013 at 10:35 AM, Harsh J ha...@cloudera.com wrote:
Moving the question
?
--
Harsh J
and DataFileReader. But how to enable
the compression for avro serialized buffer.
Thanks and Regards,
Vinod
--
Harsh J
the
order of the schema to express a different sort order, but might present
problems for schema negotiation.
What kind of problems are you describing here? Sorry if I'm not
getting it by the words schema negotiation alone.
--
Harsh J
28, 2013 at 11:27 PM, Jeremy Kahn troc...@trochee.net wrote:
Thanks for the information, Harsh. Further comments inline below:
On Thu, Mar 28, 2013 at 4:01 AM, Harsh J ha...@cloudera.com wrote:
On Thu, Mar 28, 2013 at 5:15 AM, Jeremy Kahn jer...@trochee.net wrote:
I can read ordered
()}/${outputDir}/value
/property
/configuration
/map-reduce
--
Harsh J
that someone will come
up with an interface that requires just one line of user code to achieve
append.
--- On Wed, 2/6/13, Harsh J ha...@cloudera.com wrote:
From: Harsh J ha...@cloudera.com
Subject: Re: Is it possible to append to an already existing avro file
To: user@avro.apache.org
Date
this is run, I get the stack trace included in the gist.
I've
run
out of things to try to fix this... I'd really really appreciate any
help I
can get. Thanks!
--
Harsh J
Group,
Are there any sample code/documentation available on writing Map-reduce
jobs with secondary sort using Avro data?
--
Thanks,
Ravi
--
Harsh J
,
Ravi
--
Harsh J
van der Lugt janl...@gmail.com wrote:
Good find! I'll take a look at this tomorrow, see if I can come up with a
fix.
On Sun, Aug 26, 2012 at 5:26 AM, Harsh J ha...@cloudera.com wrote:
I'm not an expert on the Avro C++ implementation, but I wonder if this
is cause of the nulls not being
();
if (n = 2) { throw avro::Exception(Union index too big); }
switch (n) {
case 0:
d.decodeNull();
v.set_null();
break;
case 1:
{
std::vector$Undefined$ vv;
avro::decode(d, vv);
v.set_array(vv);
}
break;
}
}
--
Harsh J
-14 20:13 2.0K
Is this an oversight, or should I be looking elsewhere?
-Steven Willis
--
Harsh J
what is the deflate level) and put it
to Avro and it became 38G
Why Avro is so big in size? Am I missing some size optimization?
Thanks in advance!
--
Harsh J
-Mar-2012 16:27 10M
Java-Apache (old)
Where would i use the different JAR files?
Many thanks
Regards
Saptarshi
--
Harsh J
-tp3920732p3920732.html
Sent from the Avro - Users mailing list archive at Nabble.com.
--
Harsh J
this as as opportunity to
learn Python and get up to speed.
Thanks a lot.
-Selvi
--
Harsh J
Customer Ops. Engineer, Cloudera
'gcc' failed with exit
status 1
On Tue, Jan 24, 2012 at 11:01 AM, Harsh J ha...@cloudera.com wrote:
Selvi,
Expanding on Douglas' response, if you have installed Avro's python
libraries (Simplest way to get latest stable is: easy_install avro,
or install from the distribution -- Post back
I do not understand what you're trying to achieve here.
Encoders work at the primitive level - they merely serialize a given data
structure (records, unions, for example), and not look at the schema (Notice -
you create a record with a schema, not an encoder with a schema). Decoders
could do
. Can someone point me to the right direction?
Thanks,
--
Miki
--
Harsh J
to add a field Map (TreeMap). I know that we can use Array and it works but
I would like to be able to get by key :)
Thank you,
Wei Shung
--
Harsh J
http://harshj.com
at Nabble.com.
--
Harsh J
www.harshj.com
, the install goes fine). I do not have
enough experience with Maven to know if it is a fault I'm doing or if
it is a fault with the build files related to maven. Any help with
creating eclipse project files for Avro's Java sub-project?
--
Harsh J
www.harshj.com
,
Felix
--
Harsh J
www.harshj.com
to an open DataFile for a given schema. Alternatively, you
can also write avro serialized data bytes into SequenceFiles.
I believe the Hadoop MapReduce trunk may have some good code on Avro
serialization classes and uses of that in MapReduce.
On Fri, Jan 28, 2011 at 12:22 PM, Harsh J qwertyman
= datafile.DataFileWriter(
open(OUTFILE_NAME, 'wb'),
io.DatumWriter(),
)
--
Harsh J
www.harshj.com
Sorry, minor error, not 'wb', but 'ab+'
df_writer = datafile.DataFileWriter(
open(OUTFILE_NAME, 'ab+'),
io.DatumWriter(),
)
--
Harsh J
www.harshj.com
behavior or is it a bug?
--
Harsh J
www.harshj.com
55 matches
Mail list logo
