Flume is indeed something you should look into. 'Log files' is a
simplification. Flume really handles events and yes logs are a common kind
of event but not the only one.
Regards
Bertrand
On Wed, May 7, 2014 at 4:48 AM, Alex Lee eliy...@hotmail.com wrote:
Sensors' may send tcpip data to
Flume is not just for log files, you can wire up Flume's source for this
purpose. Also there are alternative open-source solutions for data
streaming, e.g. Apache Storm or Kafka.
On Tue, May 6, 2014 at 10:48 PM, Alex Lee eliy...@hotmail.com wrote:
Sensors' may send tcpip data to server. Each
Whether you use Storm/kafka or any other realtime processing or not, you
may still need to persist the data which can be done directly to hbase from
any of these realtime system or from the source.
On Thu, May 8, 2014 at 9:25 PM, Hardik Pandya smarty.ju...@gmail.comwrote:
If I were you I would
or you can use combination of kafka http://kafka.apache.org/ +
phoenixhttp://phoenix.incubator.apache.org/
On Wed, May 7, 2014 at 8:55 PM, Azuryy Yu azury...@gmail.com wrote:
Hi Alex,
you can try Apache Flume.
On Wed, May 7, 2014 at 10:48 AM, Alex Lee eliy...@hotmail.com wrote:
Sensors'
Use the Flume.
On Wed, May 7, 2014 at 8:18 AM, Alex Lee eliy...@hotmail.com wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data
like a stream to the server, the quatity of the sensors and the data rate
of the data is high.
Firstly, how the data from tcpip can be
I will suggest that don't pipe the sensor data to the HDFS directly instead
you can have some program(either java,python etc) on the server itself to
process the incoming sensor data and writing it to the text/binary
file(don't know the data format which you are currently receiving).now you
can
If I were you I would ask following questions to get the answer
forget about for a minute and ask yourself how tcpip data are currently
being stored - in fs/rdbmbs?
hadoop is for offiline batch processing - if you are looking for real
time streaming solution - there is a storm (from linkedin)
Hi Alex,
you can try Apache Flume.
On Wed, May 7, 2014 at 10:48 AM, Alex Lee eliy...@hotmail.com wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data
like a stream to the server, the quatity of the sensors and the data rate
of the data is high.
Firstly, how the
Apache Flume isn't just 'for log files' - it is an event collection
framework and would fit your use-case.
For further questions over Flume, please ask on u...@flume.apache.org.
On Wed, May 7, 2014 at 8:18 AM, Alex Lee eliy...@hotmail.com wrote:
Sensors' may send tcpip data to server. Each
Sensors' may send tcpip data to server. Each sensor may send tcpip data like a
stream to the server, the quatity of the sensors and the data rate of the data
is high.
Firstly, how the data from tcpip can be put into hadoop. It need to do some
process and store in hbase. Does it need through
10 matches
Mail list logo
