So this is basically why the flume suggestion has come up. Flume natively acts as a syslog listener and will write files to basically anything (HDFS, Hive, HBase, S3).
On Thu, Apr 20, 2017 at 8:15 AM Michael Ridley <[email protected]> wrote: > When we say ingest from Kafka, what does that mean? I understand we can > read from Kafka to ingest into the cluster, but how will the data get to > Kafka and what data are we talking about? My understanding is that right > now the primary data sources would be Netflow and Syslog, neither of which > writes to Kafka natively so we would need something like StreamSets in the > middle. Certainly StreamSets UDP source -> Kafka would work. > > Michael > > On Wed, Apr 19, 2017 at 7:05 PM, kant kodali <[email protected]> wrote: > > > sure I guess Kafka has something called Kafka connect but may not be as > > mature as flume since I heard about this recently. > > > > On Wed, Apr 19, 2017 at 3:39 PM, Austin Leahy <[email protected]> > > wrote: > > > > > The advantage of flume or a flume Kafka hybrid is that the team doesn't > > > have to build sinks for any new source types added to the project just > > > create configs pointing to the landing pad > > > On Wed, Apr 19, 2017 at 3:31 PM kant kodali <[email protected]> > wrote: > > > > > > > What kind of benchmarks are we looking for? just throughput? since I > am > > > > assuming this is for ingestion. I haven't seen anything faster than > > Kafka > > > > and that is because of its simplicity after all publisher appends > > message > > > > to a file(so called the partition in kafka) and clients just do > > > sequential > > > > reads from a file so its a matter of disk throughput. The benchmark > > > numbers > > > > I have for Kafka is at very least 75K messages/sec where each message > > is > > > > 1KB on m4.xlarge which by default has EBS storage (EBS is > > > network-attached > > > > SSD disk). The network attached disk has a max throughput of > > > > 125MB/s(m4.xlarge has 1Gigabit) but if we were deploy it on ephemeral > > > > storage (local-SSD) and on a 10 Gigabit Network we would easily get > > 5-10X > > > > more. > > > > > > > > No idea about flume. > > > > > > > > Finally, not trying to pitch for Kafka however it is fastest I have > > seen > > > > but if someone has better numbers for flume then we should use that. > > > Also I > > > > would suspect there are benchmarks for Kafka vs Flume available > online > > > > already or we can try it with our own datasets. > > > > > > > > Thanks! > > > > > > > > On Wed, Apr 19, 2017 at 3:09 PM, Austin Leahy < > > [email protected]> > > > > wrote: > > > > > > > > > I am happy to create and test a flume source... #intelteam would > need > > > to > > > > > create the benchmark by deploying it and pointing a data source at > > > it... > > > > > since I don't have good enough volume of source data handy > > > > > On Wed, Apr 19, 2017 at 3:04 PM Ross, Alan D < > [email protected]> > > > > > wrote: > > > > > > > > > > > We discussed this in our staff meeting a bit today. I would like > > to > > > > see > > > > > > some benchmarking of different approaches (kafka, flume, etc) to > > see > > > > what > > > > > > the numbers look like. Is anyone in the community willing to > > > volunteer > > > > on > > > > > > this work? > > > > > > > > > > > > -----Original Message----- > > > > > > From: Austin Leahy [mailto:[email protected]] > > > > > > Sent: Wednesday, April 19, 2017 1:05 PM > > > > > > To: [email protected] > > > > > > Subject: Re: [Discuss] - Future plans for Spot-ingest > > > > > > > > > > > > I think Kafka is probably a red herring. It's an industry goto in > > the > > > > > > application world because because of redundancy but the type and > > > > volumes > > > > > of > > > > > > network telemetry that we are talking about here will bog kafka > > down > > > > > unless > > > > > > you dedicate really serious hardware to just the kafka > > > implementation. > > > > > It's > > > > > > essentially the next level of the problem that the team was > already > > > > > running > > > > > > into when rabbitMQ was queueing in data. > > > > > > > > > > > > On Wed, Apr 19, 2017 at 12:33 PM Mark Grover <[email protected]> > > > wrote: > > > > > > > > > > > > > On Wed, Apr 19, 2017 at 10:19 AM, Smith, Nathanael P < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Mark, > > > > > > > > > > > > > > > > just digesting the below. > > > > > > > > > > > > > > > > Backing up in my thought process, I was thinking that the > > ingest > > > > > > > > master (first point of entry into the system) would want to > put > > > the > > > > > > > > data into a standard serializable format. I was thinking that > > > > > > > > libraries (such as pyarrow in this case) could help by > writing > > > the > > > > > > > > data in parquet format early in the process. You are probably > > > > > > > > correct that at this point in time it might not be worth the > > time > > > > and > > > > > > can be kept in the backlog. > > > > > > > > That being said, I still think the master should produce data > > in > > > a > > > > > > > > standard format, what in your opinion (and I open this up of > > > course > > > > > > > > to > > > > > > > > others) would be the most logical format? > > > > > > > > the most basic would be to just keep it as a .csv. > > > > > > > > > > > > > > > > The master will likely write data to a staging directory in > > HDFS > > > > > > > > where > > > > > > > the > > > > > > > > spark streaming job will pick it up for normalization/writing > > to > > > > > > > > parquet > > > > > > > in > > > > > > > > the correct block sizes and partitions. > > > > > > > > > > > > > > > > > > > > > > Hi Nate, > > > > > > > Avro is usually preferred for such a standard format - because > it > > > > > > > asserts a schema (types, etc.) which CSV doesn't and it allows > > for > > > > > > > schema evolution which depending on the type of evolution, CSV > > may > > > or > > > > > > may not support. > > > > > > > And, that's something I have seen being done very commonly. > > > > > > > > > > > > > > Now, if the data were in Kafka before it gets to master, one > > could > > > > > > > argue that the master could just send metadata to the workers > > > (topic > > > > > > > name, partition number, offset start and end) and the workers > > could > > > > > > > read from Kafka directly. I do understand that'd be a much > > > different > > > > > > > architecture than the current one, but if you think it's a good > > > idea > > > > > > > too, we could document that, say in a JIRA, and (de-)prioritize > > it > > > > > > > (and in line with the rest of the discussion on this thread, > it's > > > not > > > > > > the top-most priority). > > > > > > > Thoughts? > > > > > > > > > > > > > > - Nathanael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Apr 17, 2017, at 1:12 PM, Mark Grover <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > > Thanks all your opinion. > > > > > > > > > > > > > > > > > > I think it's good to consider two things: > > > > > > > > > 1. What do (we think) users care about? > > > > > > > > > 2. What's the cost of changing things? > > > > > > > > > > > > > > > > > > About #1, I think users care more about what format data is > > > > > > > > > written > > > > > > > than > > > > > > > > > how the data is written. I'd argue whether that uses Hive, > > MR, > > > or > > > > > > > > > a > > > > > > > > custom > > > > > > > > > Parquet writer is not as important to them as long as we > > > maintain > > > > > > > > > data/format compatibility. > > > > > > > > > About #2, having worked on several projects, I find that > it's > > > > > > > > > rather difficult to keep up with Parquet. Even in Spark, > > there > > > > are > > > > > > > > > a few > > > > > > > > different > > > > > > > > > ways to write to Parquet - there's a regular mode, and a > > legacy > > > > > > > > > mode < > https://github.com/apache/spark/blob/master/sql/core/ > > > > > > > > src/main/scala/org/apache/spark/sql/execution/ > > > datasources/parquet/ > > > > > > > > ParquetWriteSupport.scala#L44> > > > > > > > > > which > > > > > > > > > continues to cause confusion > > > > > > > > > <https://issues.apache.org/jira/browse/SPARK-20297> till > > date. > > > > > > > > > Parquet itself is pretty dependent on Hadoop > > > > > > > > > <https://github.com/Parquet/parquet-mr/search?l=Maven+POM&; > > > > > > > > q=hadoop&type=&utf8=%E2%9C%93> > > > > > > > > > and, > > > > > > > > > just integrating it with systems with a lot of developers > > (like > > > > > > > > > Spark < > > > > https://www.google.com/webhp?sourceid=chrome-instant&ion=1&; > > > > > > > > espv=2&ie=UTF-8#q=spark+parquet+jiras>) > > > > > > > > > is still a lot of work. > > > > > > > > > > > > > > > > > > I personally think we should leverage higher level tools > like > > > > > > > > > Hive, or Spark to write data in widespread formats > (Parquet, > > > > being > > > > > > > > > a very good > > > > > > > > > example) but I personally wouldn't encourage us to manage > the > > > > > > > > > writers ourselves. > > > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > On Mon, Apr 17, 2017 at 11:44 AM, Michael Ridley > > > > > > > > > <[email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Without having given it too terribly much thought, that > > seems > > > > > > > > >> like an > > > > > > > OK > > > > > > > > >> approach. > > > > > > > > >> > > > > > > > > >> Michael > > > > > > > > >> > > > > > > > > >> On Mon, Apr 17, 2017 at 2:33 PM, Nathanael Smith < > > > > > > > [email protected]> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >>> I think the question is rather we can write the data > > > > generically > > > > > > > > >>> to > > > > > > > > HDFS > > > > > > > > >>> as parquet without the use of hive/impala? > > > > > > > > >>> > > > > > > > > >>> Today we write parquet data using the hive/mapreduce > > method. > > > > > > > > >>> As part of the redesign i’d like to use libraries for > this > > as > > > > > > > > >>> opposed > > > > > > > > to > > > > > > > > >> a > > > > > > > > >>> hadoop dependency. > > > > > > > > >>> I think it would be preferred to use the python master to > > > write > > > > > > > > >>> the > > > > > > > > data > > > > > > > > >>> into the format we want, then do normalization of the > data > > in > > > > > > > > >>> spark streaming. > > > > > > > > >>> Any thoughts? > > > > > > > > >>> > > > > > > > > >>> - Nathanael > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>>> On Apr 17, 2017, at 11:08 AM, Michael Ridley > > > > > > > > >>>> <[email protected]> > > > > > > > > >>> wrote: > > > > > > > > >>>> > > > > > > > > >>>> I had thought that the plan was to write the data in > > Parquet > > > > in > > > > > > > > >>>> HDFS ultimately. > > > > > > > > >>>> > > > > > > > > >>>> Michael > > > > > > > > >>>> > > > > > > > > >>>> On Sun, Apr 16, 2017 at 11:55 AM, kant kodali > > > > > > > > >>>> <[email protected]> > > > > > > > > >>> wrote: > > > > > > > > >>>> > > > > > > > > >>>>> Hi Mark, > > > > > > > > >>>>> > > > > > > > > >>>>> Thank you so much for hearing my argument. And I > > definetly > > > > > > > understand > > > > > > > > >>> that > > > > > > > > >>>>> you guys have bunch of things to do. My only concern is > > > that > > > > I > > > > > > > > >>>>> hope > > > > > > > > it > > > > > > > > >>>>> doesn't take too long too support other backends. For > > > example > > > > > > > > @Kenneth > > > > > > > > >>> had > > > > > > > > >>>>> given an example of LAMP stack had not moved away from > > > mysql > > > > > > > > >>>>> yet > > > > > > > > which > > > > > > > > >>>>> essentially means its probably a decade ? I see that in > > the > > > > > > > > >>>>> current architecture the results from with python > > > > > > > > >>>>> multiprocessing or Spark Streaming are written back to > > HDFS > > > > > > > > >>>>> and If so, can we write them in > > > > > > > > >>> parquet > > > > > > > > >>>>> format ? such that users should be able to plug in any > > > query > > > > > > > > >>>>> engine > > > > > > > > >> but > > > > > > > > >>>>> again I am not pushing you guys to do this right away > or > > > > > > > > >>>>> anything > > > > > > > > just > > > > > > > > >>>>> seeing if there a way for me to get started in parallel > > and > > > > if > > > > > > > > >>>>> not feasible, its fine I just wanted to share my 2 > cents > > > and > > > > I > > > > > > > > >>>>> am glad > > > > > > > my > > > > > > > > >>>>> argument is heard! > > > > > > > > >>>>> > > > > > > > > >>>>> Thanks much! > > > > > > > > >>>>> > > > > > > > > >>>>> On Fri, Apr 14, 2017 at 1:38 PM, Mark Grover < > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > >>>>> > > > > > > > > >>>>>> Hi Kant, > > > > > > > > >>>>>> Just wanted to make sure you don't feel like we are > > > ignoring > > > > > > > > >>>>>> your > > > > > > > > >>>>>> comment:-) I hear you about pluggability. > > > > > > > > >>>>>> > > > > > > > > >>>>>> The design can and should be pluggable but the project > > has > > > > > > > > >>>>>> one > > > > > > > stack > > > > > > > > >> it > > > > > > > > >>>>>> ships out of the box with, one stack that's the > default > > > > stack > > > > > > > > >>>>>> in > > > > > > > the > > > > > > > > >>>>> sense > > > > > > > > >>>>>> that it's the most tested and so on. And, for us, > that's > > > our > > > > > > > current > > > > > > > > >>>>> stack. > > > > > > > > >>>>>> If we were to take Apache Hive as an example, it > shipped > > > > (and > > > > > > > ships) > > > > > > > > >>> with > > > > > > > > >>>>>> MapReduce as the default configuration engine. At some > > > > point, > > > > > > > Apache > > > > > > > > >>> Tez > > > > > > > > >>>>>> came along and wanted Hive to run on Tez, so they > made a > > > > > > > > >>>>>> bunch of > > > > > > > > >>> things > > > > > > > > >>>>>> pluggable to run Hive on Tez (instead of the only > option > > > > > > > > >>>>>> up-until > > > > > > > > >> then: > > > > > > > > >>>>>> Hive-on-MR) and then Apache Spark came and re-used > some > > of > > > > > > > > >>>>>> that pluggability and even added some more so > > > Hive-on-Spark > > > > > > > > >>>>>> could > > > > > > > become > > > > > > > > a > > > > > > > > >>>>>> reality. In the same way, I don't think anyone > disagrees > > > > here > > > > > > > > >>>>>> that pluggabilty is a good thing but it's hard to do > > > > > > > > >>>>>> pluggability > > > > > > > right, > > > > > > > > >> and > > > > > > > > >>>>> at > > > > > > > > >>>>>> the right level, unless on has a clear use-case in > mind. > > > > > > > > >>>>>> > > > > > > > > >>>>>> As a project, we have many things to do and I > personally > > > > > > > > >>>>>> think the > > > > > > > > >>>>> biggest > > > > > > > > >>>>>> bang for the buck for us in making Spot a really solid > > and > > > > > > > > >>>>>> the > > > > > > > best > > > > > > > > >>> cyber > > > > > > > > >>>>>> security solution isn't pluggability but the things we > > are > > > > > > > > >>>>>> working > > > > > > > > on > > > > > > > > >>> - a > > > > > > > > >>>>>> better user interface, a common/unified approach to > > > storing > > > > > > > > >>>>>> and > > > > > > > > >>> modeling > > > > > > > > >>>>>> data, etc. > > > > > > > > >>>>>> > > > > > > > > >>>>>> Having said that, we are open, if it's important to > you > > or > > > > > > > > >>>>>> someone > > > > > > > > >>> else, > > > > > > > > >>>>>> we'd be happy to receive and review those patches. > > > > > > > > >>>>>> > > > > > > > > >>>>>> Thanks! > > > > > > > > >>>>>> Mark > > > > > > > > >>>>>> > > > > > > > > >>>>>> On Fri, Apr 14, 2017 at 10:14 AM, kant kodali > > > > > > > > >>>>>> <[email protected] > > > > > > > > > > > > > > > > >>>>> wrote: > > > > > > > > >>>>>> > > > > > > > > >>>>>>> Thanks Ross! and yes option C sounds good to me as > well > > > > > > > > >>>>>>> however I > > > > > > > > >> just > > > > > > > > >>>>>>> think Distributed Sql query engine and the resource > > > > manager > > > > > > > should > > > > > > > > >> be > > > > > > > > >>>>>>> pluggable. > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> On Fri, Apr 14, 2017 at 9:55 AM, Ross, Alan D < > > > > > > > > >> [email protected]> > > > > > > > > >>>>>>> wrote: > > > > > > > > >>>>>>> > > > > > > > > >>>>>>>> Option C is to use python on the front end of ingest > > > > > > > > >>>>>>>> pipeline > > > > > > > and > > > > > > > > >>>>>>>> spark/scala on the back end. > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> Option A uses python workers on the backend > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> Option B uses all scala. > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> -----Original Message----- > > > > > > > > >>>>>>>> From: kant kodali [mailto:[email protected]] > > > > > > > > >>>>>>>> Sent: Friday, April 14, 2017 9:53 AM > > > > > > > > >>>>>>>> To: [email protected] > > > > > > > > >>>>>>>> Subject: Re: [Discuss] - Future plans for > Spot-ingest > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> What is option C ? am I missing an email or > something? > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> On Fri, Apr 14, 2017 at 9:15 AM, Chokha > Palayamkottai > > < > > > > > > > > >>>>>>>> [email protected]> wrote: > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>>> +1 for Python 3.x > > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>>> On 4/14/2017 11:59 AM, Austin Leahy wrote: > > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>>>> I think that C is the strong solution, getting the > > > > ingest > > > > > > > really > > > > > > > > >>>>>>>>>> strong is going to lower barriers to adoption. > Doing > > > it > > > > > > > > >>>>>>>>>> in > > > > > > > > Python > > > > > > > > >>>>>>>>>> will open up the ingest portion of the project to > > > > include > > > > > > > > >>>>>>>>>> many > > > > > > > > >>>>> more > > > > > > > > >>>>>>>> developers. > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> Before it comes up I would like to throw the > > following > > > > on > > > > > > > > >>>>>>>>>> the > > > > > > > > >>>>>> pile... > > > > > > > > >>>>>>>>>> Major > > > > > > > > >>>>>>>>>> python projects django/flash, others are dropping > > 2.x > > > > > > > > >>>>>>>>>> support > > > > > > > in > > > > > > > > >>>>>>>>>> releases scheduled in the next 6 to 8 months. > Hadoop > > > > > > > > >>>>>>>>>> projects > > > > > > > in > > > > > > > > >>>>>>>>>> general tend to lag in modern python support, lets > > > > please > > > > > > > build > > > > > > > > >>>>> this > > > > > > > > >>>>>>>>>> in 3.5 so that we don't have to immediately > expect a > > > > > > > > >>>>>>>>>> rebuild > > > > > > > in > > > > > > > > >>>>> the > > > > > > > > >>>>>>>>>> pipeline. > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> -Vote C > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> Thanks Nate > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> Austin > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> On Fri, Apr 14, 2017 at 8:52 AM Alan Ross > > > > > > > > >>>>>>>>>> <[email protected]> > > > > > > > > >>>>> wrote: > > > > > > > > >>>>>>>>>> > > > > > > > > >>>>>>>>>> I really like option C because it gives a lot of > > > > > > > > >>>>>>>>>> flexibility > > > > > > > for > > > > > > > > >>>>>>>>>> ingest > > > > > > > > >>>>>>>>>>> (python vs scala) but still has the robust spark > > > > > > > > >>>>>>>>>>> streaming > > > > > > > > >>>>> backend > > > > > > > > >>>>>>>>>>> for performance. > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> Thanks for putting this together Nate. > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> Alan > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> On Fri, Apr 14, 2017 at 8:44 AM, Chokha > > > Palayamkottai < > > > > > > > > >>>>>>>>>>> [email protected]> wrote: > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>> I agree. We should continue making the existing > > stack > > > > > > > > >>>>>>>>>>> more > > > > > > > > >> mature > > > > > > > > >>>>>> at > > > > > > > > >>>>>>>>>>>> this point. Maybe if we have enough community > > > support > > > > > > > > >>>>>>>>>>>> we can > > > > > > > > >> add > > > > > > > > >>>>>>>>>>>> additional datastores. > > > > > > > > >>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> Chokha. > > > > > > > > >>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> On 4/14/17 11:10 AM, [email protected] wrote: > > > > > > > > >>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> Hi Kant, > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> YARN is the standard scheduler in Hadoop. If > > you're > > > > > > > > >>>>>>>>>>>>> using > > > > > > > > >>>>>>>>>>>>> Hive+Spark, then sure you'll have YARN. > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> Haven't seen any HIVE on Mesos so far. As said, > > > Spot > > > > > > > > >>>>>>>>>>>>> is > > > > > > > based > > > > > > > > >>>>> on > > > > > > > > >>>>>> a > > > > > > > > >>>>>>>>>>>>> quite standard Hadoop stack and I wouldn't > switch > > > too > > > > > > > > >>>>>>>>>>>>> many > > > > > > > > >>>>> pieces > > > > > > > > >>>>>>>> yet. > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> In most Opensource projects you start relying > on > > a > > > > > > > well-known > > > > > > > > >>>>>>>>>>>>> stack and then you begin to support other DB > > > backends > > > > > > > > >>>>>>>>>>>>> once > > > > > > > > >> it's > > > > > > > > >>>>>>>>>>>>> quite mature. Think in the loads of LAMP apps > > which > > > > > > > > >>>>>>>>>>>>> haven't > > > > > > > > >>>>> been > > > > > > > > >>>>>>>>>>>>> ported away from MySQL yet. > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> In any case, you'll need a high performance > SQL + > > > > > > > > >>>>>>>>>>>>> Massive > > > > > > > > >>>>> Storage > > > > > > > > >>>>>>>>>>>>> + Machine Learning + Massive Ingestion, and... > > ATM, > > > > > > > > >>>>>>>>>>>>> + that > > > > > > > can > > > > > > > > >> be > > > > > > > > >>>>>>>>>>>>> only provided by Hadoop. > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> Regards! > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> Kenneth > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> A 2017-04-14 12:56, kant kodali escrigué: > > > > > > > > >>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> Hi Kenneth, > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> Thanks for the response. I think you made a > > case > > > > for > > > > > > > > >>>>>>>>>>>>>> HDFS however users may want to use S3 or some > > > other > > > > > > > > >>>>>>>>>>>>>> FS in which > > > > > > > > >>>>> case > > > > > > > > >>>>>>>>>>>>>> they can use Auxilio (hoping that there are no > > > > > > > > >>>>>>>>>>>>>> changes > > > > > > > > needed > > > > > > > > >>>>>>>>>>>>>> within Spot in which case I > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> can > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> agree to that). for example, Netflix stores all > > > there > > > > > > > > >>>>>>>>>>>> data > > > > > > > > into > > > > > > > > >>>>> S3 > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> The distributed sql query engine I would say > > > should > > > > > > > > >>>>>>>>>>>>>> be > > > > > > > > >>>>> pluggable > > > > > > > > >>>>>>>>>>>>>> with whatever user may want to use and there a > > > bunch > > > > > > > > >>>>>>>>>>>>>> of > > > > > > > them > > > > > > > > >>>>> out > > > > > > > > >>>>>>>> there. > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> sure > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> Impala is better than hive but what if users are > > > > > > > > >>>>>>>>>>>> already > > > > > > > using > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> something > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> else like Drill or Presto? > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> Me personally, would not assume that users are > > > > > > > > >>>>>>>>>>>>>> willing to > > > > > > > > >>>>> deploy > > > > > > > > >>>>>>>>>>>>>> all > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> of > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> that and make their existing stack more > > complicated > > > at > > > > > > > > >>>>>>>>>>>> very > > > > > > > > >>>>> least > > > > > > > > >>>>>> I > > > > > > > > >>>>>>>>>>>>>> would > > > > > > > > >>>>>>>>>>>>>> say it is a uphill battle. Things have been > > > changing > > > > > > > rapidly > > > > > > > > >>>>> in > > > > > > > > >>>>>>>>>>>>>> Big > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> data > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> space so whatever we think is standard won't be > > > > > > > > >>>>>>>>>>>> standard > > > > > > > > >> anymore > > > > > > > > >>>>>>>>>>>> but > > > > > > > > >>>>>>>>>>>>>> importantly there shouldn't be any reason why > we > > > > > > > > >>>>>>>>>>>>>> shouldn't > > > > > > > > be > > > > > > > > >>>>>>>>>>>>>> flexible right. > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> Also I am not sure why only YARN? why not make > > > that > > > > > > > > >>>>>>>>>>>>>> also > > > > > > > > more > > > > > > > > >>>>>>>>>>>>>> flexible so users can pick Mesos or > standalone. > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> I think Flexibility is a key for a wide > adoption > > > > > > > > >>>>>>>>>>>>>> rather > > > > > > > than > > > > > > > > >>>>> the > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>> tightly > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> coupled architecture. > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> Thanks! > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> On Fri, Apr 14, 2017 at 3:12 AM, Kenneth > Peiruza > > > > > > > > >>>>>>>>>>>>>> <[email protected]> > > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> PS: you need a big data platform to be able to > > > > > > > > >>>>>>>>>>>>>> collect all > > > > > > > > >>>>> those > > > > > > > > >>>>>>>>>>>>>>> netflows > > > > > > > > >>>>>>>>>>>>>>> and logs. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Spot isn't intended for SMBs, that's clear, > > then > > > > you > > > > > > > > >>>>>>>>>>>>>>> need > > > > > > > > >>>>> loads > > > > > > > > >>>>>>>>>>>>>>> of data to get ML working properly, and > > somewhere > > > > to > > > > > > > > >>>>>>>>>>>>>>> run > > > > > > > > >>>>> those > > > > > > > > >>>>>>>>>>>>>>> algorithms. That > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> is > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> Hadoop. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Regards! > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Kenneth > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Sent from my Mi phone On kant kodali > > > > > > > > >>>>>>>>>>>>>>> <[email protected]>, Apr 14, 2017 4:04 > > > > > > > AM > > > > > > > > >>>>>> wrote: > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Hi, > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Thanks for starting this thread. Here is my > > > > feedback. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> I somehow think the architecture is too > > > complicated > > > > > > > > >>>>>>>>>>>>>>> for > > > > > > > > wide > > > > > > > > >>>>>>>>>>>>>>> adoption since it requires to install the > > > > following. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> HDFS. > > > > > > > > >>>>>>>>>>>>>>> HIVE. > > > > > > > > >>>>>>>>>>>>>>> IMPALA. > > > > > > > > >>>>>>>>>>>>>>> KAFKA. > > > > > > > > >>>>>>>>>>>>>>> SPARK (YARN). > > > > > > > > >>>>>>>>>>>>>>> YARN. > > > > > > > > >>>>>>>>>>>>>>> Zookeeper. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Currently there are way too many dependencies > > > that > > > > > > > > >>>>> discourages > > > > > > > > >>>>>>>>>>>>>>> lot of users from using it because they have > to > > > go > > > > > > > through > > > > > > > > >>>>>>>>>>>>>>> deployment of all that required software. I > > think > > > > > > > > >>>>>>>>>>>>>>> for > > > > > > > wide > > > > > > > > >>>>>>>>>>>>>>> option we should minimize the dependencies > and > > > have > > > > > > > > >>>>>>>>>>>>>>> more pluggable architecture. for example I am > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> not > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> sure why HIVE & IMPALA both are required? why > not > > > just > > > > > > > > >>>>>>>>>>>> use > > > > > > > > >> Spark > > > > > > > > >>>>>>>>>>>> SQL > > > > > > > > >>>>>>>>>>>>>>> since > > > > > > > > >>>>>>>>>>>>>>> its already dependency or say users may want > to > > > use > > > > > > > > >>>>>>>>>>>>>>> their > > > > > > > > >> own > > > > > > > > >>>>>>>>>>>>>>> distributed query engine they like such as > > Apache > > > > > > > > >>>>>>>>>>>>>>> Drill > > > > > > > or > > > > > > > > >>>>>>>>>>>>>>> something else. we should be flexible enough > to > > > > > > > > >>>>>>>>>>>>>>> provide > > > > > > > > that > > > > > > > > >>>>>>>>>>>>>>> option > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Also, I see that HDFS is used such that > > > collectors > > > > > > > > >>>>>>>>>>>>>>> can > > > > > > > > >>>>> receive > > > > > > > > >>>>>>>>>>>>>>> file path's through Kafka and be able to > read a > > > > > > > > >>>>>>>>>>>>>>> file. How > > > > > > > > >> big > > > > > > > > >>>>>>>>>>>>>>> are these files ? > > > > > > > > >>>>>>>>>>>>>>> Do we > > > > > > > > >>>>>>>>>>>>>>> really need HDFS for this? Why not provide > more > > > > ways > > > > > > > > >>>>>>>>>>>>>>> to > > > > > > > > send > > > > > > > > >>>>>>>>>>>>>>> data such as sending data directly through > > Kafka > > > or > > > > > > > > >>>>>>>>>>>>>>> say > > > > > > > > just > > > > > > > > >>>>>>>>>>>>>>> leaving up to the user to specify the file > > > location > > > > > > > > >>>>>>>>>>>>>>> as an argument to collector process > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Finally, I learnt that to generate Net flow > > data > > > > one > > > > > > > would > > > > > > > > >>>>>>>>>>>>>>> require a specific hardware. This really > means > > > > > > > > >>>>>>>>>>>>>>> Apache > > > > > > > Spot > > > > > > > > >> is > > > > > > > > >>>>>>>>>>>>>>> not meant for everyone. > > > > > > > > >>>>>>>>>>>>>>> I thought Apache Spot can be used to analyze > > the > > > > > > > > >>>>>>>>>>>>>>> network > > > > > > > > >>>>>> traffic > > > > > > > > >>>>>>>>>>>>>>> of > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>> any > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> machine but if it requires a specific hard then > I > > > > think > > > > > > > > >>>>>>>>>>>> it > > > > > > > is > > > > > > > > >>>>>>>>>>>>>>> targeted for > > > > > > > > >>>>>>>>>>>>>>> specific group of people. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> The real strength of Apache Spot should > mainly > > be > > > > > > > > >>>>>>>>>>>>>>> just > > > > > > > > >>>>>> analyzing > > > > > > > > >>>>>>>>>>>>>>> network traffic through ML. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Thanks! > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> On Thu, Apr 13, 2017 at 4:28 PM, Segerlind, > > > Nathan > > > > L > > > > > > > > >>>>>>>>>>>>>>> < [email protected]> wrote: > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> Thanks, Nate, > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Nate. > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> -----Original Message----- > > > > > > > > >>>>>>>>>>>>>>>> From: Nate Smith [mailto: > > [email protected]] > > > > > > > > >>>>>>>>>>>>>>>> Sent: Thursday, April 13, 2017 4:26 PM > > > > > > > > >>>>>>>>>>>>>>>> To: [email protected] > > > > > > > > >>>>>>>>>>>>>>>> Cc: [email protected]; > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> [email protected] > > > > > > > > >>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> Subject: Re: [Discuss] - Future plans for > > > Spot-ingest > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> I was really hoping it came through ok, Oh > > well > > > :) > > > > > > > Here’s > > > > > > > > >> an > > > > > > > > >>>>>>>>>>>>>>>> image form: > > > > > > > > >>>>>>>>>>>>>>>> http://imgur.com/a/DUDsD > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> On Apr 13, 2017, at 4:05 PM, Segerlind, > Nathan > > > L < > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> [email protected]> wrote: > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> The diagram became garbled in the text > > format. > > > > > > > > >>>>>>>>>>>>>>>>> Could you resend it as a pdf? > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> Thanks, > > > > > > > > >>>>>>>>>>>>>>>>> Nate > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> -----Original Message----- > > > > > > > > >>>>>>>>>>>>>>>>> From: Nathanael Smith > > > > > > > > >>>>>>>>>>>>>>>>> [mailto:[email protected]] > > > > > > > > >>>>>>>>>>>>>>>>> Sent: Thursday, April 13, 2017 4:01 PM > > > > > > > > >>>>>>>>>>>>>>>>> To: [email protected]; > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> [email protected]; > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> [email protected] > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> Subject: [Discuss] - Future plans for > > > Spot-ingest > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> How would you like to see Spot-ingest > change? > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> A. continue development on the Python > > > > > > > > >>>>>>>>>>>>>>>>> Master/Worker > > > > > > > with > > > > > > > > >>>>>> focus > > > > > > > > >>>>>>>>>>>>>>>>> on > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> performance / error handling / logging B. > > > Develop > > > > > > > > >>>>>>>>>>>>>>>> Scala > > > > > > > > >>>>> based > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> ingest to > > > > > > > > >>>>>>>>>>>>>>> be > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> inline with code base from ingest, ml, to OA > > (UI > > > > to > > > > > > > > >> continue > > > > > > > > >>>>>>>>>>>>>>>> being > > > > > > > > >>>>>>>>>>>>>>>> ipython/JS) C. Python ingest Worker with > Scala > > > > > > > > >>>>>>>>>>>>>>>> based > > > > > > > Spark > > > > > > > > >>>>>> code > > > > > > > > >>>>>>>>>>>>>>>> for normalization and input into DB > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> Including the high level diagram: > > > > > > > > >>>>>>>>>>>>>>>>> +----------------------------- > > > > > > > > >>>>> ------------------------------ > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> -------------------------------+ > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | +--------------------------+ > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +-----------------+ | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | Master | A. B. C. > > > > > > > > >>>>>>> | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Worker | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | A. Python > > +---------------+ > > > > > > A. > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | A. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Python | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | B. Scala | > > | > > > > > > > > >>>>>>> +-------------> > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +----+ | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | C. Python | > > | > > > > | > > > > > > > > >>>>>>> | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | +---^------+---------------+ > > | > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +-----------------+ | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | | > > | > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | | > > | > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | +Note--------------+ > > | > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +-----------------+ | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | |Running on a | > > | > > > > | > > > > > > > > >>>>>>> | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Spark > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Streaming | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | |worker node in | > > | > > > > | > > > > > > > > >> B. > > > > > > > > >>>>>> C. > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | B. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Scala | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | |the Hadoop cluster| > > | > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +--------> C. > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Scala +-+ | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | +------------------+ > > | > > > > | > > > > > > > | > > > > > > > > >>>>>>> | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | A.| > > | > > > > | > > > > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +-----------------+ | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | B.| > > | > > > > | > > > > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | C.| > > | > > > > | > > > > > > > | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | +----------------------+ > > > > > > > > +-v------+----+----+-+ > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +--------------v--v-+ | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | | | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | Local FS: | | > hdfs > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> Hive / Impala | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | - Binary/Text | | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> - Parquet - | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | Log files - | | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | | | | > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> | | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> | +----------------------+ > > > > > > > > +--------------------+ > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> +-------------------+ | > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> +----------------------------- > > > > > > > > >>>>> ------------------------------ > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> -------------------------------+ > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> Please let me know your thoughts, > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> - Nathanael > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > >>>>>>>>>>>> > > > > > > > > >>>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>> > > > > > > > > >>>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> -- > > > > > > > > >>>> Michael Ridley <[email protected]> > > > > > > > > >>>> office: (650) 352-1337 > > > > > > > > >>>> mobile: (571) 438-2420 > > > > > > > > >>>> Senior Solutions Architect > > > > > > > > >>>> Cloudera, Inc. > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> -- > > > > > > > > >> Michael Ridley <[email protected]> > > > > > > > > >> office: (650) 352-1337 > > > > > > > > >> mobile: (571) 438-2420 > > > > > > > > >> Senior Solutions Architect > > > > > > > > >> Cloudera, Inc. > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Michael Ridley <[email protected]> > office: (650) 352-1337 > mobile: (571) 438-2420 > Senior Solutions Architect > Cloudera, Inc. >
