[
https://issues.apache.org/jira/browse/FLUME-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15663538#comment-15663538
]
PIPELINE edited comment on FLUME-3022 at 11/15/16 9:22 AM:
-----------------------------------------------------------
batchSize = 100
deserializer.maxLineLength = 20480
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
more slow...
----------------------------------------------------
may be jar conflict........
was (Author: pipe):
batchSize = 100
deserializer.maxLineLength = 20480
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
more slow...
14 Nov 2016 17:04:47,718 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:227)
- Start checkpoint for
/data/data/flume/source/appMain/file-channel/checkpoint/checkpo
int, elements to sync = 33100
14 Nov 2016 17:04:48,069 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:252) -
Updating checkpoint metadata: logWriteOrderID: 1479114298634, queueSize: 26700,
queueHead
: 6398
14 Nov 2016 17:04:48,464 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.Log.writeCheckpoint:1052) - Updated checkpoint
for file: /data/data/flume/source/appMain/file-channel/data/log-1 position:
15737863 lo
gWriteOrderID: 1479114298634
14 Nov 2016 17:05:18,514 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:227)
- Start checkpoint for
/data/data/flume/source/appMain/file-channel/checkpoint/checkpo
int, elements to sync = 49155
14 Nov 2016 17:05:18,949 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:252) -
Updating checkpoint metadata: logWriteOrderID: 1479114348280, queueSize: 55145,
queueHead
: 16753
14 Nov 2016 17:05:19,994 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.Log.writeCheckpoint:1052) - Updated checkpoint
for file: /data/data/flume/source/appMain/file-channel/data/log-1 position:
33681561 lo
gWriteOrderID: 1479114348280
14 Nov 2016 17:05:50,023 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:227)
- Start checkpoint for
/data/data/flume/source/appMain/file-channel/checkpoint/checkpo
int, elements to sync = 52245
14 Nov 2016 17:05:50,485 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:252) -
Updating checkpoint metadata: logWriteOrderID: 1479114401050, queueSize: 82500,
queueHead
: 29198
14 Nov 2016 17:05:50,959 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.Log.writeCheckpoint:1052) - Updated checkpoint
for file: /data/data/flume/source/appMain/file-channel/data/log-1 position:
52128469 lo
gWriteOrderID: 1479114401050
14 Nov 2016 17:06:20,967 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:227)
- Start checkpoint for
/data/data/flume/source/appMain/file-channel/checkpoint/checkpo
int, elements to sync = 48967
14 Nov 2016 17:06:21,364 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:252) -
Updating checkpoint metadata: logWriteOrderID: 1479114450506, queueSize:
107533, queueHea
d: 41165
14 Nov 2016 17:06:21,819 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.Log.writeCheckpoint:1052) - Updated checkpoint
for file: /data/data/flume/source/appMain/file-channel/data/log-1 position:
69840880 lo
gWriteOrderID: 1479114450506
14 Nov 2016 17:06:51,868 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:227)
- Start checkpoint for
/data/data/flume/source/appMain/file-channel/checkpoint/checkpo
int, elements to sync = 50933
14 Nov 2016 17:06:52,398 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:252) -
Updating checkpoint metadata: logWriteOrderID: 1479114501949, queueSize:
134200, queueHea
d: 53298
14 Nov 2016 17:06:52,930 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.Log.writeCheckpoint:1052) - Updated checkpoint
for file: /data/data/flume/source/appMain/file-channel/data/log-1 position:
88399115 lo
gWriteOrderID: 1479114501949
14 Nov 2016 17:07:22,963 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.beginCheckpoint:227)
- Start checkpoint for
/data/data/flume/source/appMain/file-channel/checkpoint/checkpo
int, elements to sync = 50600
14 Nov 2016 17:07:23,361 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:252) -
Updating checkpoint metadata: logWriteOrderID: 1479114553056, queueSize:
161400, queueHea
d: 64998
14 Nov 2016 17:07:23,798 INFO [Log-BackgroundWorker-appMainFileChanel]
(org.apache.flume.channel.file.Log.writeCheckpoint:1052) - Updated checkpoint
for file: /data/data/flume/source/appMain/file-channel/data/log-1 position:
107129401 l
ogWriteOrderID: 1479114553056
> SpoolDirSource can not work well on Chinese log, and translating more slow
> than 1.5.2 or 1.6.0 with the same conf.
> ------------------------------------------------------------------------------------------------------------------
>
> Key: FLUME-3022
> URL: https://issues.apache.org/jira/browse/FLUME-3022
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.7.0
> Environment: jdk1.7
> Reporter: PIPELINE
>
> Cause 1.5.2 or 1.6.0 has spoolDirSource bug with emoji, i tried 1.7.0.
> well ,it resolved emoji bug, but may create new bugs. such as:
> 1. can not work well on Chinese log files.
> 2.translating more slow than 1.5.2 or 1.6.0 with the same conf.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
