[ https://issues.apache.org/jira/browse/BEAM-9188?focusedWorklogId=378530&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378530 ]
ASF GitHub Bot logged work on BEAM-9188: ---------------------------------------- Author: ASF GitHub Bot Created on: 28/Jan/20 22:26 Start Date: 28/Jan/20 22:26 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10701: [BEAM-9188] CassandraIO split performance improvement - cache size of the table URL: https://github.com/apache/beam/pull/10701#discussion_r372092162 ########## File path: sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java ########## @@ -508,26 +518,30 @@ private static long getNumSplits( @Override public long getEstimatedSizeBytes(PipelineOptions pipelineOptions) { - try (Cluster cluster = - getCluster( - spec.hosts(), - spec.port(), - spec.username(), - spec.password(), - spec.localDc(), - spec.consistencyLevel())) { - if (isMurmur3Partitioner(cluster)) { - try { - List<TokenRange> tokenRanges = - getTokenRanges(cluster, spec.keyspace().get(), spec.table().get()); - return getEstimatedSizeBytesFromTokenRanges(tokenRanges); - } catch (Exception e) { - LOG.warn("Can't estimate the size", e); + if (estimatedSize != null) { + return estimatedSize; + } else { + try (Cluster cluster = + getCluster( + spec.hosts(), + spec.port(), + spec.username(), + spec.password(), + spec.localDc(), + spec.consistencyLevel())) { + if (isMurmur3Partitioner(cluster)) { + try { + List<TokenRange> tokenRanges = + getTokenRanges(cluster, spec.keyspace().get(), spec.table().get()); + return getEstimatedSizeBytesFromTokenRanges(tokenRanges); Review comment: Can we also cache the size here in case `getEstimateSizeBytes` is called more than once? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 378530) Time Spent: 1.5h (was: 1h 20m) > Improving speed of splitting for Custom Sources > ----------------------------------------------- > > Key: BEAM-9188 > URL: https://issues.apache.org/jira/browse/BEAM-9188 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow > Reporter: Radosław Stankiewicz > Assignee: Radosław Stankiewicz > Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > At this moment Custom Source in being split and serialized in sequence. If > there are many splits, it takes time to process all splits. > > Example: it takes 2s to calculate size and serialize CassandraSource due to > connection setup and teardown. With 100+ splits, it's a lot of time spent in > 1 worker. -- This message was sent by Atlassian Jira (v8.3.4#803005)