[jira] [Comment Edited] (IMPALA-11677) FireInsertEvents function can be very slow for tables with large number of partitions.
[
https://issues.apache.org/jira/browse/IMPALA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620713#comment-17620713
]
Qihong Jiang edited comment on IMPALA-11677 at 10/20/22 5:05 AM:
-
Hello !, [~csringhofer] I'm only using non-transactional tables right now and
it's equally slow. I tried using the Bulk API last week, but the improvement
was very small. Then I referenced the code in impala3 and modified it to be an
asynchronous call. The execution speed is greatly improved, but I don't know if
there is any risk.
{code:java}
public static List fireInsertEvents(MetaStoreClient msClient,
TableInsertEventInfo insertEventInfo, String dbName, String tableName) {
if (!insertEventInfo.isTransactional()) {
LOG.info("fire the insert events asynchronously.");
ExecutorService fireInsertEventThread =
Executors.newSingleThreadExecutor();
CompletableFuture.runAsync(() -> {
try {
fireInsertEventHelper(msClient.getHiveClient(),
insertEventInfo.getInsertEventReqData(),
insertEventInfo.getInsertEventPartVals(), dbName,
tableName);
} catch(Exception e) {
LOG.error("failed to async call fireInsertEventHelper");
} finally {
msClient.close();
LOG.info("fire the insert events asynchronously end.");
}
}, fireInsertEventThread)
.thenRun(() -> fireInsertEventThread.shutdown());
} else {
Stopwatch sw = Stopwatch.createStarted();
try {
fireInsertTransactionalEventHelper(msClient.getHiveClient(),
insertEventInfo, dbName, tableName);
} catch (Exception e) {
LOG.error("Failed to fire insert event. Some tables might not be"
+ " refreshed on other impala clusters.", e);
} finally {
LOG.info("Time taken to fire insert events on table {}.{}: {} msec",
dbName,
tableName, sw.stop().elapsed(TimeUnit.MILLISECONDS));
msClient.close();
}
} return Collections.emptyList();
}{code}
I am an impala newbie. I hope to get your guidance. Thank you!
was (Author: JIRAUSER289149):
Hello !, [~csringhofer] I'm only using non-transactional tables right now and
it's equally slow. I tried using the Bulk API last week, but the improvement
was very small. Then I referenced the code in impala3 and modified it to be an
asynchronous call. The execution speed is greatly improved, but I don't know if
there is any risk.
{code:java}
public static List fireInsertEvents(MetaStoreClient msClient,
TableInsertEventInfo insertEventInfo, String dbName, String tableName) {
if (!insertEventInfo.isTransactional()) {
LOG.info("fire the insert events asynchronously.");
ExecutorService fireInsertEventThread =
Executors.newSingleThreadExecutor();
CompletableFuture.runAsync(() -> {
try {
fireInsertEventHelper(msClient.getHiveClient(),
insertEventInfo.getInsertEventReqData(),
insertEventInfo.getInsertEventPartVals(), dbName,
tableName);
} catch(Exception e) {
LOG.error("failed to async call fireInsertEventHelper");
} finally {
msClient.close();
LOG.info("fire the insert events asynchronously end.");
}
}, fireInsertEventThread)
.thenRun(() -> fireInsertEventThread.shutdown());
} else {
Stopwatch sw = Stopwatch.createStarted();
try {
fireInsertTransactionalEventHelper(msClient.getHiveClient(),
insertEventInfo, dbName, tableName);
} catch (Exception e) {
LOG.error("Failed to fire insert event. Some tables might not be"
+ " refreshed on other impala clusters.", e);
} finally {
LOG.info("Time taken to fire insert events on table {}.{}: {} msec",
dbName,
tableName, sw.stop().elapsed(TimeUnit.MILLISECONDS));
msClient.close();
}
} return Collections.emptyList();
}{code}
I am not an expert in impala. I hope to get your guidance. Thank you!
> FireInsertEvents function can be very slow for tables with large number of
> partitions.
> --
>
> Key: IMPALA-11677
> URL: https://issues.apache.org/jira/browse/IMPALA-11677
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Affects Versions: Impala 4.1.0
>Reporter: Qihong Jiang
>Assignee: Qihong Jiang
>Priority: Major
>
> In src
[jira] [Comment Edited] (IMPALA-11677) FireInsertEvents function can be very slow for tables with large number of partitions.
[
https://issues.apache.org/jira/browse/IMPALA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620713#comment-17620713
]
Qihong Jiang edited comment on IMPALA-11677 at 10/20/22 5:04 AM:
-
Hello !, [~csringhofer] I'm only using non-transactional tables right now and
it's equally slow. I tried using the Bulk API last week, but the improvement
was very small. Then I referenced the code in impala3 and modified it to be an
asynchronous call. The execution speed is greatly improved, but I don't know if
there is any risk.
{code:java}
public static List fireInsertEvents(MetaStoreClient msClient,
TableInsertEventInfo insertEventInfo, String dbName, String tableName) {
if (!insertEventInfo.isTransactional()) {
LOG.info("fire the insert events asynchronously.");
ExecutorService fireInsertEventThread =
Executors.newSingleThreadExecutor();
CompletableFuture.runAsync(() -> {
try {
fireInsertEventHelper(msClient.getHiveClient(),
insertEventInfo.getInsertEventReqData(),
insertEventInfo.getInsertEventPartVals(), dbName,
tableName);
} catch(Exception e) {
LOG.error("failed to async call fireInsertEventHelper");
} finally {
msClient.close();
LOG.info("fire the insert events asynchronously end.");
}
}, fireInsertEventThread)
.thenRun(() -> fireInsertEventThread.shutdown());
} else {
Stopwatch sw = Stopwatch.createStarted();
try {
fireInsertTransactionalEventHelper(msClient.getHiveClient(),
insertEventInfo, dbName, tableName);
} catch (Exception e) {
LOG.error("Failed to fire insert event. Some tables might not be"
+ " refreshed on other impala clusters.", e);
} finally {
LOG.info("Time taken to fire insert events on table {}.{}: {} msec",
dbName,
tableName, sw.stop().elapsed(TimeUnit.MILLISECONDS));
msClient.close();
}
} return Collections.emptyList();
}{code}
I am not an expert in impala. I hope to get your guidance. Thank you!
was (Author: JIRAUSER289149):
Hello !, [~csringhofer] I'm only using non-transactional tables right now and
it's equally slow. I tried using the Bulk API last week, but the improvement
was very small. Then I referenced the code in impala3 and modified it to be an
asynchronous call. The execution speed is greatly improved, but I don't know if
there is any risk.
{code:java}
public static List fireInsertEvents(MetaStoreClient msClient,
TableInsertEventInfo insertEventInfo, String dbName, String tableName) {
if (!insertEventInfo.isTransactional()) {
LOG.info("fire the insert events asynchronously.");
ExecutorService fireInsertEventThread =
Executors.newSingleThreadExecutor();
CompletableFuture.runAsync(() -> {
try {
fireInsertEventHelper(msClient.getHiveClient(),
insertEventInfo.getInsertEventReqData(),
insertEventInfo.getInsertEventPartVals(), dbName,
tableName);
} catch(Exception e) {
LOG.error("failed to async call fireInsertEventHelper");
} }, fireInsertEventThread)
.thenRun(() -> {
LOG.info("fire the insert events asynchronously end.");
msClient.close();
fireInsertEventThread.shutdown();
});
} else {
Stopwatch sw = Stopwatch.createStarted();
try {
fireInsertTransactionalEventHelper(msClient.getHiveClient(),
insertEventInfo, dbName, tableName);
} catch (Exception e) {
LOG.error("Failed to fire insert event. Some tables might not be"
+ " refreshed on other impala clusters.", e);
} finally {
LOG.info("Time taken to fire insert events on table {}.{}: {} msec",
dbName,
tableName, sw.stop().elapsed(TimeUnit.MILLISECONDS));
msClient.close();
}
} return Collections.emptyList();
}{code}
I am not an expert in impala. I hope to get your guidance. Thank you!
> FireInsertEvents function can be very slow for tables with large number of
> partitions.
> --
>
> Key: IMPALA-11677
> URL: https://issues.apache.org/jira/browse/IMPALA-11677
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Affects Versions: Impala 4.1.0
>Reporter: Qihong Jiang
>Assignee: Qihong Jiang
>P
[jira] [Commented] (IMPALA-11677) FireInsertEvents function can be very slow for tables with large number of partitions.
[
https://issues.apache.org/jira/browse/IMPALA-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620713#comment-17620713
]
Qihong Jiang commented on IMPALA-11677:
---
Hello !, [~csringhofer] I'm only using non-transactional tables right now and
it's equally slow. I tried using the Bulk API last week, but the improvement
was very small. Then I referenced the code in impala3 and modified it to be an
asynchronous call. The execution speed is greatly improved, but I don't know if
there is any risk.
{code:java}
public static List fireInsertEvents(MetaStoreClient msClient,
TableInsertEventInfo insertEventInfo, String dbName, String tableName) {
if (!insertEventInfo.isTransactional()) {
LOG.info("fire the insert events asynchronously.");
ExecutorService fireInsertEventThread =
Executors.newSingleThreadExecutor();
CompletableFuture.runAsync(() -> {
try {
fireInsertEventHelper(msClient.getHiveClient(),
insertEventInfo.getInsertEventReqData(),
insertEventInfo.getInsertEventPartVals(), dbName,
tableName);
} catch(Exception e) {
LOG.error("failed to async call fireInsertEventHelper");
} }, fireInsertEventThread)
.thenRun(() -> {
LOG.info("fire the insert events asynchronously end.");
msClient.close();
fireInsertEventThread.shutdown();
});
} else {
Stopwatch sw = Stopwatch.createStarted();
try {
fireInsertTransactionalEventHelper(msClient.getHiveClient(),
insertEventInfo, dbName, tableName);
} catch (Exception e) {
LOG.error("Failed to fire insert event. Some tables might not be"
+ " refreshed on other impala clusters.", e);
} finally {
LOG.info("Time taken to fire insert events on table {}.{}: {} msec",
dbName,
tableName, sw.stop().elapsed(TimeUnit.MILLISECONDS));
msClient.close();
}
} return Collections.emptyList();
}{code}
I am not an expert in impala. I hope to get your guidance. Thank you!
> FireInsertEvents function can be very slow for tables with large number of
> partitions.
> --
>
> Key: IMPALA-11677
> URL: https://issues.apache.org/jira/browse/IMPALA-11677
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Affects Versions: Impala 4.1.0
>Reporter: Qihong Jiang
>Assignee: Qihong Jiang
>Priority: Major
>
> In src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java.
> fireInsertEvents function can be very slow for tables with large number of
> partitions. So we should use asynchronous calls.Just like in impala-3.x
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-11677) FireInsertEvents function can be very slow for tables with large number of partitions.
Qihong Jiang created IMPALA-11677: - Summary: FireInsertEvents function can be very slow for tables with large number of partitions. Key: IMPALA-11677 URL: https://issues.apache.org/jira/browse/IMPALA-11677 Project: IMPALA Issue Type: Improvement Components: Catalog Affects Versions: Impala 4.1.0 Reporter: Qihong Jiang In src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java. fireInsertEvents function can be very slow for tables with large number of partitions. So we should use asynchronous calls.Just like in impala-3.x -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-11677) FireInsertEvents function can be very slow for tables with large number of partitions.
Qihong Jiang created IMPALA-11677: - Summary: FireInsertEvents function can be very slow for tables with large number of partitions. Key: IMPALA-11677 URL: https://issues.apache.org/jira/browse/IMPALA-11677 Project: IMPALA Issue Type: Improvement Components: Catalog Affects Versions: Impala 4.1.0 Reporter: Qihong Jiang In src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java. fireInsertEvents function can be very slow for tables with large number of partitions. So we should use asynchronous calls.Just like in impala-3.x -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-27610) PrometheusPushGatewayReporter doesn't support basic authentication
Qihong Jiang created FLINK-27610: Summary: PrometheusPushGatewayReporter doesn't support basic authentication Key: FLINK-27610 URL: https://issues.apache.org/jira/browse/FLINK-27610 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Reporter: Qihong Jiang PrometheusPushGatewayReporter doesn't support basic authentication. I would like to add two configurations (username,password) to support access to pushGateway which requires authentication -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27610) PrometheusPushGatewayReporter doesn't support basic authentication
Qihong Jiang created FLINK-27610: Summary: PrometheusPushGatewayReporter doesn't support basic authentication Key: FLINK-27610 URL: https://issues.apache.org/jira/browse/FLINK-27610 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Reporter: Qihong Jiang PrometheusPushGatewayReporter doesn't support basic authentication. I would like to add two configurations (username,password) to support access to pushGateway which requires authentication -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (YARN-10851) Tez session close does not interrupt yarn's async thread
Qihong Wu created YARN-10851:
Summary: Tez session close does not interrupt yarn's async thread
Key: YARN-10851
URL: https://issues.apache.org/jira/browse/YARN-10851
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.10.1, 2.8.5
Environment: On an HA cluster, where RM1 is not the active RM
Yarn of version 2.8.5 and is configured with Tez
Reporter: Qihong Wu
Attachments: hive.log
Hi, I want to ask for the expertise knowledge on the yarn behavior when
handling `InterruptedIOException`.
The issue occurs on a HA cluster, where RM1 is NOT the active RM. Therefore, if
the yarn request made to RM1 failed, the RM failover should happen. However, if
an interrupted exception is thrown when connecting to RM1, the thread should
try to [bail
out|https://dzone.com/articles/how-to-handle-the-interruptedexception] as soon
as possible to [respect interrupt
request|https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html#shutdownNow--],
rather than moving on to another RM.
But I found my application (hive) after throwing `InterruptedIOException` when
trying to connect with RM1 failed, continuing to RM2. I want to know how does
yarn handle InterruptedIOException, shouldn't the async thread gets interrupted
and shutdown when tez close() triggered interrupt request?
*The reproduction step is:*
1. In an HA cluster which uses yarn of version 2.8.5 and is configured with Tez
2. Make sure RM1 is not the active RM by checking `yarn rmadmin
-getAllServiceState`. It it is, manually [transition RM2 as active
RM|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html#Admin_commands].
3. Apply failover-retry properties to yarn-site.xml
{quote}
yarn.client.failover-retries
4
yarn.client.failover-retries-on-socket-timeouts
4
yarn.client.failover-max-attempts
4
{quote}
4. Run a simple application to yarn-client (for example, a simple hive DDL
command)
{quote}hive --hiveconf hive.root.logger=TRACE,console -e "create table tez_test
(id int, name string);"
{quote}
5. Find from application's log (for example, hive.log), you can find
`RetryInvocationHandler` has captured the `InterruptedIOException` when request
was talking over rm1, but the thread didn't bail out immediately, but continue
moving to rm2.
*More information:*
The interrupted exception is triggered via via
[TezSessionState#close|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L689]
and
[Future#cancel|https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html#cancel-boolean-].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10851) Tez session close does not interrupt yarn's async thread
Qihong Wu created YARN-10851:
Summary: Tez session close does not interrupt yarn's async thread
Key: YARN-10851
URL: https://issues.apache.org/jira/browse/YARN-10851
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.10.1, 2.8.5
Environment: On an HA cluster, where RM1 is not the active RM
Yarn of version 2.8.5 and is configured with Tez
Reporter: Qihong Wu
Attachments: hive.log
Hi, I want to ask for the expertise knowledge on the yarn behavior when
handling `InterruptedIOException`.
The issue occurs on a HA cluster, where RM1 is NOT the active RM. Therefore, if
the yarn request made to RM1 failed, the RM failover should happen. However, if
an interrupted exception is thrown when connecting to RM1, the thread should
try to [bail
out|https://dzone.com/articles/how-to-handle-the-interruptedexception] as soon
as possible to [respect interrupt
request|https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html#shutdownNow--],
rather than moving on to another RM.
But I found my application (hive) after throwing `InterruptedIOException` when
trying to connect with RM1 failed, continuing to RM2. I want to know how does
yarn handle InterruptedIOException, shouldn't the async thread gets interrupted
and shutdown when tez close() triggered interrupt request?
*The reproduction step is:*
1. In an HA cluster which uses yarn of version 2.8.5 and is configured with Tez
2. Make sure RM1 is not the active RM by checking `yarn rmadmin
-getAllServiceState`. It it is, manually [transition RM2 as active
RM|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html#Admin_commands].
3. Apply failover-retry properties to yarn-site.xml
{quote}
yarn.client.failover-retries
4
yarn.client.failover-retries-on-socket-timeouts
4
yarn.client.failover-max-attempts
4
{quote}
4. Run a simple application to yarn-client (for example, a simple hive DDL
command)
{quote}hive --hiveconf hive.root.logger=TRACE,console -e "create table tez_test
(id int, name string);"
{quote}
5. Find from application's log (for example, hive.log), you can find
`RetryInvocationHandler` has captured the `InterruptedIOException` when request
was talking over rm1, but the thread didn't bail out immediately, but continue
moving to rm2.
*More information:*
The interrupted exception is triggered via via
[TezSessionState#close|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L689]
and
[Future#cancel|https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Future.html#cancel-boolean-].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (ZEPPELIN-5149) NPE in RemoteInterpreterServer#interpret
Qihong Wu created ZEPPELIN-5149:
---
Summary: NPE in RemoteInterpreterServer#interpret
Key: ZEPPELIN-5149
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5149
Project: Zeppelin
Issue Type: Bug
Reporter: Qihong Wu
This is the same bug in https://issues.apache.org/jira/browse/ZEPPELIN-4829#.
Reproduce step is to run any query with %livy.sql interpreter in zeppelin, for
example
%livy.sql
show databases
The bug is caused by two NPEs and the solution in ZEPPELIN-4829 only resolved
the first NPE. With v0.9.0-preview2, I still can reproduce the same error with
log
{quote}
ERROR [2020-12-01 07:28:58,404] (\{pool-2-thread-1}
ProcessFunction.java[process]:47) - Internal error processing interpretERROR
[2020-12-01 07:28:58,404] (\{pool-2-thread-1} ProcessFunction.java[process]:47)
- Internal error processing interpretjava.lang.NullPointerException at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.interpret(RemoteInterpreterServer.java:624)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1646)
at
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$interpret.getResult(RemoteInterpreterService.java:1626)
at
org.apache.zeppelin.shaded.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at
org.apache.zeppelin.shaded.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.zeppelin.shaded.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{quote}
The unresolved NPE is
[https://github.com/apache/zeppelin/blob/v0.9.0-preview2/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java#L624]
I don't see the error in v0.8.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARTEMIS-2806) deployQueue missing address argument
Qihong Xu created ARTEMIS-2806: -- Summary: deployQueue missing address argument Key: ARTEMIS-2806 URL: https://issues.apache.org/jira/browse/ARTEMIS-2806 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.12.0 Reporter: Qihong Xu In ActiveMQServerControlImpl, deployQueue method is missing address argument which results in creating non-matched addresses and queues. (For example, using deployQueue to create a subscriber A_0 under existing address A, it finally returns a new topic A_0 with a subscriber A_0 ) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932616#comment-16932616
]
Qihong Chen commented on KAFKA-7500:
Hi [~ryannedolan], Thanks for your quick response (y)(y)(y)
Now I know what "primary" means. According to blog [A look inside Kafka
MirrorMaker2|https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/] by
Renu Tewari,
{quote}In MM2 only one connect cluster is needed for all the cross-cluster
replications between a pair of datacenters. Now if we simply take a Kafka
source and sink connector and deploy them in tandem to do replication, the data
would need to hop through an intermediate Kafka cluster. MM2 avoids this
unnecessary data copying by a direct passthrough from source to sink.
{quote}
This is exactly what I want! Do you have release schedule for *SinkConnector*,
and *direct passthrough from source to sink* feature?
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.4.0
>Reporter: Ryanne Dolan
>Assignee: Manikumar
>Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931743#comment-16931743 ] Qihong Chen edited comment on KAFKA-7500 at 9/17/19 7:22 PM: - Hi [~ryannedolan], just listened your Kafka Power Chat talk, Thanks! Have a follow up question about the last question (from me) you answered in the talk. You said you prefer dedicated MM2 cluster over running MM2 in connect cluster since you can use less number of clusters to do replications among multiple Kafka clusters. But there's no REST Api for a dedicated MM2 cluster that can provide the status of the replication streams, nor updating the replication configuration. Any changes to the configuration meaning update the config files and restart all MM2 instances, is that right? Or I missed it that dedicated MM2 cluster does provide REST API for admin and monitoring, if so, where is it? If my understanding is correct, we can archive the same thing with MM2 in connect cluster. Assume there are 3 Kafka clusters: A, B, and C. Set up a connect cluster against C (meaning all topics for connectors' data and states go to cluster C), then set up MM2 connectors to replicate data and metadata A -> B and B -> A. If this is correct, we can use the Kafka cluster C plus the connect cluster that running against Kafka cluster C to replicate data among more Kafka clusters, like A, B, and D, and even more. Of course, this needs more complicated configuration, which requires deeper understanding how the MM2 connectors work. In this scenario, the connect cluster provides REST API to admin and monitoring all the connectors. This will be useful for people can't use Stream Replication Manager from Cloudera or Kafka replicator from Confluent for some reason. Is this right? was (Author: qihong): Hi [~ryannedolan], just listened your Kafka Power Chat talk, Thanks! Have a follow up question about the last question (from me) you answered in the talk. You said you prefer dedicated MM2 cluster over running MM2 in connect cluster since you can use less number of clusters to do replications among multiple Kafka clusters. But there's no REST Api for a dedicated MM2 cluster that can provide the status of the replication streams, nor updating the replication configuration. Any changes to the configuration meaning update the config files and restart all MM2 instances, is that right? Or I missed it that dedicated MM2 cluster does provide REST API for admin and monitoring, if so, where is it? If my understanding is correct, we can archive the same thing with MM2 in connect cluster. Assume there are 3 Kafka clusters: A, B, and C. Set up a connect cluster against C (meaning all topics for connectors' data and states go to cluster C), then set up MM2 connectors to replicate data and metadata A -> B and B -> A. If this is correct, we can use the Kafka cluster C plus the connect cluster that running against Kafka cluster C to replicate data among more Kafka clusters, like A, B, and D, and even more. Of course, this needs more complicated configuration, which requires deeper understanding how the MM2 connectors work. In this scenario, the connect cluster provides REST API to admin and monitoring all the connectors. This will be useful for people can't use Stream Replication Manager from Cloudera or Kafka replicator from Confluent for some reason. Is this right? > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931743#comment-16931743 ] Qihong Chen commented on KAFKA-7500: Hi [~ryannedolan], just listened your Kafka Power Chat talk, Thanks! Have a follow up question about the last question (from me) you answered in the talk. You said you prefer dedicated MM2 cluster over running MM2 in connect cluster since you can use less number of clusters to do replications among multiple Kafka clusters. But there's no REST Api for a dedicated MM2 cluster that can provide the status of the replication streams, nor updating the replication configuration. Any changes to the configuration meaning update the config files and restart all MM2 instances, is that right? Or I missed it that dedicated MM2 cluster does provide REST API for admin and monitoring, if so, where is it? If my understanding is correct, we can archive the same thing with MM2 in connect cluster. Assume there are 3 Kafka clusters: A, B, and C. Set up a connect cluster against C (meaning all topics for connectors' data and states go to cluster C), then set up MM2 connectors to replicate data and metadata A -> B and B -> A. If this is correct, we can use the Kafka cluster C plus the connect cluster that running against Kafka cluster C to replicate data among more Kafka clusters, like A, B, and D, and even more. Of course, this needs more complicated configuration, which requires deeper understanding how the MM2 connectors work. In this scenario, the connect cluster provides REST API to admin and monitoring all the connectors. This will be useful for people can't use Stream Replication Manager from Cloudera or Kafka replicator from Confluent for some reason. Is this right? > MirrorMaker 2.0 (KIP-382) > - > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Manikumar >Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (KAFKA-7500) MirrorMaker 2.0 (KIP-382)
[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927863#comment-16927863
]
Qihong Chen edited comment on KAFKA-7500 at 9/11/19 6:13 PM:
-
[~ryannedolan] Thanks for all your efforts on MM2. Appreciate your input on the
following questions.
I'm trying to replicate topics from one cluster to another, include topic data
and related consumers' offset. But I only see the topic data was replicated,
not consumer offsets.
Here's my mm2.properties
{code:java}
clusters = dr1, dr2
dr1.bootstrap.servers = 10.1.0.4:9092,10.1.0.5:9092,10.1.0.6:9092
dr2.bootstrap.servers = 10.2.0.4:9092,10.2.0.5:9092,10.2.0.6:9092
# only allow replication dr1 -> dr2
dr1->dr2.enabled = true
dr1->dr2.topics = test.*
dr1->dr2.groups = test.*
dr1->dr2.emit.heartbeats.enabled = false
dr2->dr1.enabled = false
dr2->dr1.emit.heartbeats.enabled = false
{code}
Here's how I started MM2 cluster (dr2 as the nearby cluster)
{code:java}
nohup bin/connect-mirror-maker.sh mm2.properties --clusters dr2 > mm2.log 2>&1 &
{code}
On dr1, there's topic *test1*, and consumer group *test1grp* for topic test1.
On dr2, I found following topics
{code:java}
__consumer_offsets
dr1.checkpoints.internal
dr1.test1
heartbeats
mm2-configs.dr1.internal
mm2-offsets.dr1.internal
mm2-status.dr1.internal
{code}
But couldn't find any consumer groups on dr2 related to consumer group
*test1grp*.
Could you please let me know in detail how to migrate consumer group *test1grp*
from dr1 to dr2?, i.e. what command(s) need to run to set up the offset for
*test1grp* on dr2 before consume topic *dr1.test1* ?
By the way, how to set up and run this in a Kafka connect cluster? i.e., how to
set up
MirrorSourceConnector, MirrorCheckpointConnector in a connect cluster? Is
there document about this?
was (Author: qihong):
[~ryannedolan] Thanks for all your efforts on MM2. Appreciate your input on the
following questions.
I'm trying to replicate topics from one cluster to another, include topic data
and related consumers' offset. But I only see the topic data was replicated,
not consumer offsets.
Here's my mm2.properties
{code:java}
clusters = dr1, dr2
dr1.bootstrap.servers = 10.1.0.4:9092,10.1.0.5:9092,10.1.0.6:9092
dr2.bootstrap.servers = 10.2.0.4:9092,10.2.0.5:9092,10.2.0.6:9092
# only allow replication dr1 -> dr2
dr1->dr2.enabled = true
dr1->dr2.topics = test.*
dr1->dr2.groups = test.*
dr1->dr2.emit.heartbeats.enabled = false
dr2->dr1.enabled = false
dr2->dr1.emit.heartbeats.enabled = false
{code}
Here's how I started MM2 cluster (dr2 as the nearby cluster)
{code:java}
nohup bin/connect-mirror-maker.sh mm2.properties --clusters dr2 > mm2.log 2>&1 &
{code}
On dr1, there's topic *test1*, and consumer group *test1grp* for topic test1.
On dr2, I found following topics
{code:java}
__consumer_offsets
dr1.checkpoints.internal
dr1.test1
heartbeats
mm2-configs.dr1.internal
mm2-offsets.dr1.internal
mm2-status.dr1.internal
{code}
But couldn't find any consumer groups on dr2 related consumer group *test1grp*.
Could you please let me know in detail how to migrate consumer group
*test1grp* from dr1 to dr2?, i.e. what command(s) need to run to set up the
offset for *test1grp* on dr2 before consume topic *dr1.test1* ?
By the way, how to set up and run this in a Kafka connect cluster? i.e., how to
set up
MirrorSourceConnector, MirrorCheckpointConnector in a connect cluster? Is
there document about this?
> MirrorMaker 2.0 (KIP-382)
> -
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
>Affects Versions: 2.4.0
>Reporter: Ryanne Dolan
>Assignee: Manikumar
>Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
Is Xerces-c 3.2.2 slower than 3.1.4 part 2
Due to the limitation of the email size, I have to send the rest of attachment in a separate email. -- Qihong Wang Senior Software Developer ANSYS, Inc. www.ansys.com Southpointe 2600 ANSYS Drive Canonsburg, PA 15317-9565 Phone:724-820-3966 Fax:724-820-3816 E-Mail: qihong.w...@ansys.com
[jira] [Created] (ARTEMIS-2251) Large messages might not be deleted when server crashed
Qihong Xu created ARTEMIS-2251: -- Summary: Large messages might not be deleted when server crashed Key: ARTEMIS-2251 URL: https://issues.apache.org/jira/browse/ARTEMIS-2251 Project: ActiveMQ Artemis Issue Type: Bug Reporter: Qihong Xu When deleting large messages, artemis will use storePendingLargeMessage to insert a temporary record in journal for reload, in case server crashed and large messages stayed forever. But in storePendingLargeMessage that appendAddRecord inserts records asynchronously. In this way there are potential risks that tasks in executor get lost due to server crash, which may lead to undeletable large messages. To solve this problem a Boolean is added to storePendingLargeMessage so that it will be forced to use SimpleWaitIOCallback in delete situation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2214) ARTEMIS-2214 Cache durable in PagedReference
[ https://issues.apache.org/jira/browse/ARTEMIS-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ARTEMIS-2214: --- Summary: ARTEMIS-2214 Cache durable in PagedReference (was: Cache durable in PagedReference to avoid blocks in consuming paged messages) > ARTEMIS-2214 Cache durable in PagedReference > - > > Key: ARTEMIS-2214 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > Attachments: stacks.txt > > > We recently performed a test on artemis broker and found a severe performance > issue. > When paged messages are being consumed, decrementMetrics in > QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are > durable or not. In this way queue will be locked for a long time because page > may be GCed and need to be reload entirely. Other operations rely on queue > will be blocked at this time, which cause a significant TPS drop. Detailed > stacks are attached below. > This also happens when consumer is closed and messages are pushed back to the > queue, artemis will check priority on return if these messages are paged. > To solve the issue, durable and priority need to be cached in PagedReference > just like messageID, transactionID and so on. I have applied a patch to fix > the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2216) Use a specific executor for pageSyncTimer
[ https://issues.apache.org/jira/browse/ARTEMIS-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ARTEMIS-2216: --- Description: Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot. We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput. To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker. Broker: 4C/8G/500G SSD Producer: 200 threads, non-transactional send Consumer 200 threads, transactional receive Message text size: 100-200 bytes randomly AddressFullPolicy: PAGE Test result： | |Only Send TPS|Only Receive TPS|Send TPS| |Original ver|38k|33k|3k/30k| |Modified ver|38k|34k|30k/12.5k| The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable. was: Improve paging throughput by using a specific executor for pageSyncTimer Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot. We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput. To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker. Broker: 4C/8G/500G SSD Producer: 200 threads, non-transactional send Consumer 200 threads, transactional receive Message text size: 100-200 bytes randomly AddressFullPolicy: PAGE Test result： | |Only Send TPS|Only Receive TPS|Send TPS| |Original ver|38k|33k|3k/30k| |Modified ver|38k|34k|30k/12.5k| The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable. > Use a specific executor for pageSyncTimer > - > > Key: ARTEMIS-2216 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2216 > Project: ActiveMQ Artemis > Issue Type: Improvement >Affects Versions: 2.6.3 > Reporter: Qihong Xu >Priority: Major > > Improving throughput on paging mode is one of our concerns since our cluster > uses paging a lot. > We found that pageSyncTimer in PagingStoreImpl shared the same executor with > pageCursorProvider from thread pool. In heavy load scenario like hundreds of > consumers receiving messages simultaneously, it became difficult for > pageSyncTimer to get the executor due to race condition. Therefore page sync > was delayed and producers suffered low throughput. > > To achieve higher performance we assign a specific executor to pageSyncTimer > to avoid racing. And we run a small-scale test on a single modified broker. > > Broker: 4C/8G/500G SSD > Producer: 200 threads, non-transactional send > Consumer 200 threads, transactional receive > Message text size: 100-200 bytes randomly > AddressFullPolicy: PAGE > > Test result： > | |Only Send TPS|Only Receive TPS|Send TPS| > |Original ver|38k|33k|3k/30k| > |Modified ver|38k|34k|30k/12.5k| > > The chart above shows that on modified broker send TPS improves from “poor” > to “extremely fast”, while receive TPS drops from “extremely fast” to > “not-bad” under heavy load. Considering consumer systems usually have a long > processing chain after receiving messages, we don’t need too fast receive > TPS. Instead, we w
[jira] [Created] (ARTEMIS-2216) Use a specific executor for pageSyncTimer
Qihong Xu created ARTEMIS-2216: -- Summary: Use a specific executor for pageSyncTimer Key: ARTEMIS-2216 URL: https://issues.apache.org/jira/browse/ARTEMIS-2216 Project: ActiveMQ Artemis Issue Type: Improvement Affects Versions: 2.6.3 Reporter: Qihong Xu Improve paging throughput by using a specific executor for pageSyncTimer Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot. We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput. To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker. Broker: 4C/8G/500G SSD Producer: 200 threads, non-transactional send Consumer 200 threads, transactional receive Message text size: 100-200 bytes randomly AddressFullPolicy: PAGE Test result： | |Only Send TPS|Only Receive TPS|Send TPS| |Original ver|38k|33k|3k/30k| |Modified ver|38k|34k|30k/12.5k| The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2214) Cache durable in PagedReference to avoid blocks in consuming paged messages
Qihong Xu created ARTEMIS-2214: -- Summary: Cache durable in PagedReference to avoid blocks in consuming paged messages Key: ARTEMIS-2214 URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 Project: ActiveMQ Artemis Issue Type: Bug Components: Broker Affects Versions: 2.6.3 Reporter: Qihong Xu We recently performed a test on artemis broker and found a severe performance issue. When paged messages are being consumed, decrementMetrics in QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are durable or not. In this way queue will be locked for a long time because page may be GCed and need to be reload entirely. Other operations rely on queue will be blocked at this time, which cause a significant TPS drop. Detailed stacks are attached below. This also happens when consumer is closed and messages are pushed back to the queue, artemis will check priority on return if these messages are paged. To solve the issue, durable and priority need to be cached in PagedReference just like messageID, transactionID and so on. I have applied a patch to fix the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2214) Cache durable in PagedReference to avoid blocks in consuming paged messages
[ https://issues.apache.org/jira/browse/ARTEMIS-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ARTEMIS-2214: --- Attachment: (was: 0001-Add-durable-and-priority-to-pagedReference.patch) > Cache durable in PagedReference to avoid blocks in consuming paged > messages > > > Key: ARTEMIS-2214 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > Attachments: stacks.txt > > > We recently performed a test on artemis broker and found a severe performance > issue. > When paged messages are being consumed, decrementMetrics in > QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are > durable or not. In this way queue will be locked for a long time because page > may be GCed and need to be reload entirely. Other operations rely on queue > will be blocked at this time, which cause a significant TPS drop. Detailed > stacks are attached below. > This also happens when consumer is closed and messages are pushed back to the > queue, artemis will check priority on return if these messages are paged. > To solve the issue, durable and priority need to be cached in PagedReference > just like messageID, transactionID and so on. I have applied a patch to fix > the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2214) Cache durable in PagedReference to avoid blocks in consuming paged messages
[ https://issues.apache.org/jira/browse/ARTEMIS-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ARTEMIS-2214: --- Attachment: 0001-Add-durable-and-priority-to-pagedReference.patch > Cache durable in PagedReference to avoid blocks in consuming paged > messages > > > Key: ARTEMIS-2214 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > Attachments: 0001-Add-durable-and-priority-to-pagedReference.patch, > stacks.txt > > > We recently performed a test on artemis broker and found a severe performance > issue. > When paged messages are being consumed, decrementMetrics in > QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are > durable or not. In this way queue will be locked for a long time because page > may be GCed and need to be reload entirely. Other operations rely on queue > will be blocked at this time, which cause a significant TPS drop. Detailed > stacks are attached below. > This also happens when consumer is closed and messages are pushed back to the > queue, artemis will check priority on return if these messages are paged. > To solve the issue, durable and priority need to be cached in PagedReference > just like messageID, transactionID and so on. I have applied a patch to fix > the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2214) Cache durable in PagedReference to avoid blocks in consuming paged messages
[ https://issues.apache.org/jira/browse/ARTEMIS-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ARTEMIS-2214: --- Attachment: stacks.txt > Cache durable in PagedReference to avoid blocks in consuming paged > messages > > > Key: ARTEMIS-2214 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2214 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.3 >Reporter: Qihong Xu >Priority: Major > Attachments: stacks.txt > > > We recently performed a test on artemis broker and found a severe performance > issue. > When paged messages are being consumed, decrementMetrics in > QueuePendingMessageMetrics will try to ‘getMessage’ to check whether they are > durable or not. In this way queue will be locked for a long time because page > may be GCed and need to be reload entirely. Other operations rely on queue > will be blocked at this time, which cause a significant TPS drop. Detailed > stacks are attached below. > This also happens when consumer is closed and messages are pushed back to the > queue, artemis will check priority on return if these messages are paged. > To solve the issue, durable and priority need to be cached in PagedReference > just like messageID, transactionID and so on. I have applied a patch to fix > the issue. Any review is appreciated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
I got an issue, when testing command like this :qemu–system–aarch64 -kernel uimage -smp 2...
The whole command , qemu–system–aarch64 -kernel uimage -M vexpress -a15 -cpu cortex-a57 -smp 2 -m256 --nographic The issue is that when running into shell, i didn't find two cpus. The parameter "-smp 2",didn't work. Does qemu support arch arm -v8 when using the parameter(-smp)? From jqh mobile phone. ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
[jira] [Commented] (ARTEMIS-1700) Server stopped responding and killed itself while exiting paging state
[ https://issues.apache.org/jira/browse/ARTEMIS-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375225#comment-16375225 ] Qihong Xu commented on ARTEMIS-1700: [~nigro@gmail.com] Yes, we just use the default setting here. > Server stopped responding and killed itself while exiting paging state > -- > > Key: ARTEMIS-1700 > URL: https://issues.apache.org/jira/browse/ARTEMIS-1700 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.4.0 >Reporter: Qihong Xu >Priority: Major > Attachments: artemis.log > > > We are currently experiencing this error while running stress test on artemis. > > Basic configuration: > 1 broker ,1 topic, pub-sub mode. > Journal type = MAPPED. > Threadpool max size = 60. > > In order to test the throughput of artemis we use 300 producers and 300 > consumers. However we found that sometimes when artemis exit paging state, it > will stop responding and kill itself. This situatuion happened on some > specific servers. > > Details can be found in attached dump file. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-1700) Server stopped responding and killed itself while exiting paging state
Qihong Xu created ARTEMIS-1700: -- Summary: Server stopped responding and killed itself while exiting paging state Key: ARTEMIS-1700 URL: https://issues.apache.org/jira/browse/ARTEMIS-1700 Project: ActiveMQ Artemis Issue Type: Bug Components: Broker Affects Versions: 2.4.0 Reporter: Qihong Xu Attachments: artemis.log We are currently experiencing this error while running stress test on artemis. Basic configuration: 1 broker ,1 topic, pub-sub mode. Journal type = MAPPED. Threadpool max size = 60. In order to test the throughput of artemis we use 300 producers and 300 consumers. However we found that sometimes when artemis exit paging state, it will stop responding and kill itself. This situatuion happened on some specific servers. Details can be found in attached dump file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-2964) "Conf" command returns dataDir and dataLogDir opposingly
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ZOOKEEPER-2964: - Summary: "Conf" command returns dataDir and dataLogDir opposingly (was: dataDir and dataLogDir are printed opposingly) > "Conf" command returns dataDir and dataLogDir opposingly > > > Key: ZOOKEEPER-2964 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2964 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.3 >Reporter: Qihong Xu >Priority: Minor > Attachments: ZOOKEEPER-2964.patch > > > I foung a bug that "conf" command would return dataDir and dataLogDir > opposingly. > This bug only exists in versions newer than 3.5. I only found dumpConf in > [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] > prints these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths > are not affected and server function is ok. > I made a small patch to fix this bug. Any review is appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (ZOOKEEPER-2964) dataDir and dataLogDir are printed opposingly
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ZOOKEEPER-2964: - Attachment: ZOOKEEPER-2964.patch > dataDir and dataLogDir are printed opposingly > - > > Key: ZOOKEEPER-2964 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2964 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.3 >Reporter: Qihong Xu >Priority: Minor > Attachments: ZOOKEEPER-2964.patch > > > I foung a bug that "conf" command would return dataDir and dataLogDir > opposingly. > This bug only exists in versions newer than 3.5. I only found dumpConf in > [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] > prints these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths > are not affected and server function is ok. > I made a small patch to fix this bug. Any review is appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (ZOOKEEPER-2964) dataDir and dataLogDir are printed opposingly
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ZOOKEEPER-2964: - Description: I foung a bug that "conf" command would return dataDir and dataLogDir opposingly. This bug only exists in versions newer than 3.5. I only found dumpConf in [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] prints these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths are not affected and server function is ok. I made a small patch to fix this bug. Any review is appreciated. was: I foung a bug that "conf" command would return dataDir and dataLogDir opposingly. This bug only exists in versions newer than 3.5. I only found dumpConf in [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] print these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths are not affected and server function is ok. > dataDir and dataLogDir are printed opposingly > - > > Key: ZOOKEEPER-2964 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2964 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.5.3 >Reporter: Qihong Xu >Priority: Minor > > I foung a bug that "conf" command would return dataDir and dataLogDir > opposingly. > This bug only exists in versions newer than 3.5. I only found dumpConf in > [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] > prints these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths > are not affected and server function is ok. > I made a small patch to fix this bug. Any review is appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (ZOOKEEPER-2964) dataDir and dataLogDir are printed opposingly
Qihong Xu created ZOOKEEPER-2964: Summary: dataDir and dataLogDir are printed opposingly Key: ZOOKEEPER-2964 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2964 Project: ZooKeeper Issue Type: Bug Components: server Affects Versions: 3.5.3 Reporter: Qihong Xu Priority: Minor I foung a bug that "conf" command would return dataDir and dataLogDir opposingly. This bug only exists in versions newer than 3.5. I only found dumpConf in [ZookeeperServer.java|https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L188] print these two paths opposingly. Unlike ZOOKEEPER-2960, the actual paths are not affected and server function is ok. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (ZOOKEEPER-2927) Local session reconnect validation not forward to leader
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Xu updated ZOOKEEPER-2927: - Description: When zookeeper quorum recovers from shutdown/crash, a client with a local session will reconnect to a random server in quorum. If this random-chosen server is not leader and does not own the local session previously, it will forward this session to leader for validation. And then if this is a global session, leader will update its owner, if not, leader adds Boolean false to packet and does nothing. Since our system involves mostly local session and has a large amount of connections, this procedure may be redundant and add potential pressure to leader. Is this reasonable for the reconnect scenario that local session validation not forward to leader, instead return by follower directly? was: When zookeeper quorum recovers from shutdown/crash, a client with a local session will reconnect to a random server in quorum. If this random-chosen server is not leader and does not own the local session previously, it will forward this session to leader for validation. And then if this is a global session, leader will update its owner, if not, leader adds Boolean false to packet and does nothing. Since our system involves mostly local session and has a large amount of connections, this procedure may be redundant and add potential pressure to leader. Is this reasonable for the reconnect scenario that local session does not forward to leader, instead return by follower directly? > Local session reconnect validation not forward to leader > > > Key: ZOOKEEPER-2927 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2927 > Project: ZooKeeper > Issue Type: Improvement > Components: java client, quorum, server >Affects Versions: 3.5.3 > Environment: configuration management system based on zookeeper 3.5.3 >Reporter: Qihong Xu >Priority: Minor > > When zookeeper quorum recovers from shutdown/crash, a client with a local > session will reconnect to a random server in quorum. If this random-chosen > server is not leader and does not own the local session previously, it will > forward this session to leader for validation. And then if this is a global > session, leader will update its owner, if not, leader adds Boolean false to > packet and does nothing. > Since our system involves mostly local session and has a large amount of > connections, this procedure may be redundant and add potential pressure to > leader. Is this reasonable for the reconnect scenario that local session > validation not forward to leader, instead return by follower directly? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: GSoC 2015 Students: "Pencils Down" and Final Evaluation Dates Approaching
Hi Dr. Jiang, Yes, the code have been submitted. Thanks for your kindly reminder. regards Qihong On Tue, Sep 22, 2015 at 10:32 AM, Ying Jiang <jpz6311...@gmail.com> wrote: > Hi Qihong, > > Have you submitted the code sample? If not, you'd better do it right > now, or you won't get the certificate. > > Best regards, > Ying Jiang > > > -- Forwarded message -- > From: 'Carol Smith' via Google Summer of Code Students List > <google-summer-of-code-students-l...@googlegroups.com> > Date: Tue, Sep 22, 2015 at 3:22 AM > Subject: Re: GSoC 2015 Students: "Pencils Down" and Final Evaluation > Dates Approaching > To: GSoC students <google-summer-of-code-students-l...@googlegroups.com> > > > Hi GSoC 2015 students, > > On Mon, Aug 10, 2015 at 8:41 AM, Carol Smith <car...@google.com> wrote: >> >> >> I have also updated the code submission guidelines for the program this >> year. Code submissions for passing students can happen any time after 28 >> August if you have passed your final evaluations. They are due by 25 >> September. Please look the guidelines over and let me know if you have any >> questions. One question students ask a lot is what portion of their code to >> submit if they made changes to an existing code base or their code interacts >> a lot with a system they didn't write. The answer is: use your best >> discretion. Submit the code that makes the most sense from a user's >> perspective. > > > This is a reminder that the soft deadline for submitting your GSoC > project code is this coming Friday, 25 September. If you would like to > be included in our first round of shipments of t-shirts and > certificates you need to submit your code by then. We do not consider > your project "complete" until you have submitted this code. > > Thanks, > Carol > > -- > You received this message because you are subscribed to the Google > Groups "Google Summer of Code Students List" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to google-summer-of-code-students-list+unsubscr...@googlegroups.com. > To post to this group, send email to > google-summer-of-code-students-l...@googlegroups.com. > Visit this group at > http://groups.google.com/group/google-summer-of-code-students-list. > For more options, visit https://groups.google.com/d/optout.
incubator-geode git commit: GEODE-77: improve restart locator error handling on corrupted state file
Repository: incubator-geode
Updated Branches:
refs/heads/feature/GEODE-77 c61fe3466 - 7fd67b883
GEODE-77: improve restart locator error handling on corrupted state file
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/7fd67b88
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/7fd67b88
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/7fd67b88
Branch: refs/heads/feature/GEODE-77
Commit: 7fd67b8836b1c0ff95034a9a6f25901e805354f2
Parents: c61fe34
Author: Qihong Chen qc...@pivotal.io
Authored: Wed Aug 19 15:12:16 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Wed Aug 19 15:13:06 2015 -0700
--
gemfire-assembly/build.gradle | 2 +-
.../distributed/internal/InternalLocator.java | 7 +-
.../membership/gms/locator/GMSLocator.java | 66 ++-
.../gemfire/internal/i18n/LocalizedStrings.java | 2 +-
.../gms/locator/GMSLocatorJUnitTest.java| 87
5 files changed, 118 insertions(+), 46 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/7fd67b88/gemfire-assembly/build.gradle
--
diff --git a/gemfire-assembly/build.gradle b/gemfire-assembly/build.gradle
index 7e7dcb3..be655e2 100755
--- a/gemfire-assembly/build.gradle
+++ b/gemfire-assembly/build.gradle
@@ -112,7 +112,7 @@ def cp = {
it.contains('spring-shell') ||
it.contains('snappy-java') ||
it.contains('hbase') ||
- it.contains('jgroups')
+ it.contains('jgroups') ||
it.contains('netty')
}.join(' ')
}
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/7fd67b88/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/InternalLocator.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/InternalLocator.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/InternalLocator.java
index f649713..32dee46 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/InternalLocator.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/InternalLocator.java
@@ -24,6 +24,7 @@ import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
+import com.gemstone.gemfire.InternalGemFireException;
import org.apache.logging.log4j.Logger;
import com.gemstone.gemfire.CancelException;
@@ -1066,7 +1067,7 @@ public class InternalLocator extends Locator implements
ConnectListener {
ThreadGroup group = LoggingThreadGroup.createThreadGroup(Locator restart
thread group);
this.restartThread = new Thread(group, Location services restart thread)
{
public void run() {
-boolean restarted;
+boolean restarted = false;
try {
restarted = attemptReconnect();
logger.info(attemptReconnect returned {}, restarted);
@@ -1074,6 +1075,10 @@ public class InternalLocator extends Locator implements
ConnectListener {
logger.info(attempt to restart location services was interrupted,
e);
} catch (IOException e) {
logger.info(attempt to restart location services terminated, e);
+} finally {
+ if (! restarted) {
+stoppedForReconnect = false;
+ }
}
InternalLocator.this.restartThread = null;
}
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/7fd67b88/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/locator/GMSLocator.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/locator/GMSLocator.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/locator/GMSLocator.java
index a988dec..dd4ac51 100755
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/locator/GMSLocator.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/locator/GMSLocator.java
@@ -14,6 +14,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Set;
+import com.gemstone.gemfire.InternalGemFireException;
import org.apache.logging.log4j.Logger;
import com.gemstone.gemfire.DataSerializer;
@@ -30,16 +31,16 @@ import
com.gemstone.gemfire.distributed.internal.membership.gms.Services;
import
com.gemstone.gemfire.distributed.internal.membership.gms.interfaces.Locator;
import
com.gemstone.gemfire.distributed.internal.membership.gms.mgr.GMSMembershipManager
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi Andy,
Thank for your comments! I can understand your idea of A/ B/ C/. But
there'a problem in practice.
In brief:
For A/, at the server side, always call
QueryExecutionBase::execConstructDataset()
For B/, the problem is, there're overlaps between DEF.rdfOffer and
DEF.quadsOffer, e.g. TriX, TriXxml and JSONLD.
For C/, e.g. if the Accept lang is JSONLD, which should be written out
by RDFDataMgr, model or dataset? Note that, the server side doesn't
know which it's called from the client side,
QueryEngineHTTP::exeConstructTriples() or exeConstructQuads().
Also at the client side, if
QueryEngineHTTP::exeConstructTriples()/exeConstructQuads() gets the
content type of, e.g. JSONLD, it doesn't know whether it is a model or
dataset.
So, that's why I introduce the DEF.pureRdfOffer to distinguish the
triple format that is not a quad format. In my way, both the sever
side and the client side separate model and dataset. It's kind of ugly
in code. Any more suggestion?
regards,
Qihong
On Mon, Aug 17, 2015 at 10:49 PM, Andy Seaborne a...@apache.org wrote:
Thanks for the clarification. I had made a combined version to start
testing and hopefully it's a close approximation of the intended
deliverables.
[ Ying - how's your testing going? ]
A few things about the pull requests so far:
0/ More tests in TestQuery in Fuseki:
For example, this should show up:
1/ QueryEngineHttp.execConstructDataset is not implemented.
2/ SPARQL_Query:
This line
if ( ! rdfMediaType.getType().equals(*) ) {
means that only Accept: * will trigger dataset results.
then in ResponseDataset
MediaType i = ConNeg.chooseContentType(request, DEF.quadsOffer,
DEF.acceptNQuads) ;
will always choose TriG because the accept is * (output= works but that
is overriding content negotiation).
There is no way to ask for a specific syntax (n-quads, trig, JSON-LD) using
Accept:
e.g. Accept: application/n-quads
3/ ResponseDataset is copy of ResponseModel yet the only differences (after
reformatting) are different data values and
RDFDataMgr.write(out, dataset, lang) ;
---
RDFDataMgr.write(out, model, lang) ;
It is not good to have copied code because it makes maintenance harder.
(2) and (3) can be addressed by
A/ SPARQL_Query:
For CONSTRUCT, always work in datasets; execConstructDataset().
No need for mention of Models. if it's a triples CONSTRUCT, treating as a
dataset will simply put the results in to the default graph.
QueryExecutionBase::execConstructQuads
Following on from that, treating datasets as a natural extension of models.
There is no need to test for extended syntax. If it's strict SPARQL 1.1, a
dataset will just have only a default graph.
B/ Content negotiate for a combined DEF.rdfOffer+DEF.quadsOffer (I don't
underatand DEF.pureRdfOffer -- N-triples is a standard).
C/ If it's a triple format (test the Lang),
RDFDataMgr.write(out, dataset.getDefaultModel(), lang) ;
otherwise:
RDFDataMgr.write(out, dataset, lang) ;
Andy
Re: svn commit: r1696184 - /jena/site/trunk/content/documentation/query/construct-quad.mdtext
Hi Dr. Jiang Thank you! I'll refine the doc following your instructions. regards, Qihong On Tue, Aug 18, 2015 at 12:21 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi Qihong, You can improve the doc: 1) The grammar table collapses. I can see a | in the first row, which needs to be escaped. Also, a table header is required. 2) All of the special symbols (e.g. code, string, expression ) should be marked. You can refer to other docs in Jena website, e.g property_paths.html 3) Add possible links in the doc, e.g. Fuseki (doc), ExampleConstructQuads.javad(code) 4) some indent problems in the code fragment 5) Add ARQ documentation index at the end of the page, just like the other ARQ docs. Last but not least, make sure make changes based on the latest source doc in svn (Andy modified it): http://svn.apache.org/repos/asf/jena/site/trunk/content/documentation/query/construct-quad.mdtext Best regards, Ying Jiang On Mon, Aug 17, 2015 at 4:52 PM, Andy Seaborne a...@apache.org wrote: On 17/08/15 00:36, jpz6311...@apache.org wrote: Author: jpz6311whu Date: Sun Aug 16 23:36:58 2015 New Revision: 1696184 URL: http://svn.apache.org/r1696184 Log: compose doc for construct-quad.mdtext Modified: jena/site/trunk/content/documentation/query/construct-quad.mdtext Please check when making changes. The generated page was blank. I have fixed this (it seems that converting the content to UTF-8 did the trick) as well as some formatting issues I noticed. I do not promise all formatting issues are fixed. Andy
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi Andy, Let me clarify PR 89 and PR 91: PR 89: confidencesun/jena/master - apache/jena/master confidencesun/jena/master contains the commits of the 1st half part of the project. It only involves jena/arq. PR 91: confidencesun/jena/JENA-491 - apache/jena/JENA-491-construct-quad confidencesun/jena/JENA-491 contains the commits of 1st and 2nd parts of the project, i.e. that's everything. Because the 2nd part of Fuseki support depends on the 1st part, so I merged confidencesun/jena/master into confidencesun/jena/JENA-491 in the first place, and then added the Fuseki support. ** If confidencesun/jena/JENA-491 contains only the commits of the 2nd part, the java maven projects won't compile. ** If you want to check only the 2nd part, just compare confidencesun/jena/master with confidencesun/jena/JENA-491, like this: https://github.com/confidencesun/jena/compare/master...confidencesun:JENA-491 You can see that, there're 5 changed files, touching both arq and fuseki: In jena-arq (client side of the Fuseki support): QueryEngineHTTP.java ( implement execQuads() and execConstructDataset()) In jena-fuseki-core (sever side): DEF.java, ResponseDataset.java, SPARQL_Query.java, TestQuery.java The implementation of the 2nd part is not complex. I don't see much code that is necessary to be tested. I'll see what I can do with the tests of different MIME types for the content negotiation. Thanks! Finally, PR 91 contains everything to be merged into apache/jena/master, and PR 89 can be ignored. Or, you can merge PR 89, and then go with PR 91. Hope it's clear. regards, Qihong On Sun, Aug 16, 2015 at 9:14 PM, Andy Seaborne a...@apache.org wrote: Hi - there are some problems with PR 91. PR 91 contains changes to jena-arq as well as jena-fuseki-core. Does PR 91 replace PR 89 or does PR 91 contain some out-of-date update now in PR 89? Or something else? ( Line 1 of PR 91 as a diff is: diff --git a/jena-arq/Grammar/arq.jj b/jena-arq/Grammar/arq.jj ) Please produce a PR which is only the changes to fuseki, and should eb applied after the ARQ changes. Also: I can only see one new test in TestQuery.java - I was expecting more such as using different MIME types for the content negotiation. Andy On 15/08/15 10:21, Qihong Lin wrote: Hi, Please check my updates of the 3 tasks blow: 1) jena/arq: in master branch (to be merged first) In PR 89, I added some more tests in TestAPI and TS_Script. The bug of construct quad projection on exeConstructTriples() has been fixed, with the tests of 4 cases (commit 03a321f). 2) jena/fuseki: in JENA-491 branch In PR 91, see commit 0eb28d8. You can try out org.apache.jena.fuseki.TestQuery.query_construct_01()/query_construct_quad_01() 3) doc: in jena web page (wiki) I can see the staging page [1]. But when I try the Improve this Page with [1], it redirects me to [2]. I can not find construct_quad.mdtext under [2]. I also try to open [3], with Not Found error. Could you help me please? regards, Qihong [1] http://jena.staging.apache.org/documentation/query/construct-quad.html [2] https://cms.apache.org/jena/wc/edit/anonymous-LWMAVL/trunk/ [3] http://jena.apache.org/documentation/query/construct-quad.html
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi,
Please check my updates of the 3 tasks blow:
1) jena/arq: in master branch (to be merged first)
In PR 89, I added some more tests in TestAPI and TS_Script. The bug of
construct quad projection on exeConstructTriples() has been fixed,
with the tests of 4 cases (commit 03a321f).
2) jena/fuseki: in JENA-491 branch
In PR 91, see commit 0eb28d8. You can try out
org.apache.jena.fuseki.TestQuery.query_construct_01()/query_construct_quad_01()
3) doc: in jena web page (wiki)
I can see the staging page [1]. But when I try the Improve this Page
with [1], it redirects me to [2]. I can not find construct_quad.mdtext
under [2]. I also try to open [3], with Not Found error. Could you
help me please?
regards,
Qihong
[1] http://jena.staging.apache.org/documentation/query/construct-quad.html
[2] https://cms.apache.org/jena/wc/edit/anonymous-LWMAVL/trunk/
[3] http://jena.apache.org/documentation/query/construct-quad.html
On Tue, Aug 11, 2015 at 10:35 AM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
You may have received the reminder email from Google that the final
evaluation of the GSoC project is approaching. The soft deadline is
17 August, with the hard one on 21 August. It means there're only 2
weeks left for your remaining work. The tests of arq need improving,
as Andy points out. You can also deliver the Fuseki part in the other
branch for review in the mean time. Meanwhile, the documentation can
be composed in jena web page (wiki). In order to save time, I suggest
the 3 parts of the work go on simultaneously:
1) jena/arq: in master branch (to be merged first)
2) jena/fuseki: in JENA-491 branch
3) doc: in jena web page (wiki)
Best regards,
Ying Jiang
On Mon, Aug 10, 2015 at 9:58 PM, Andy Seaborne a...@apache.org wrote:
I've managed to integrate the latest updates.
The scripted ones look, at first glance OK, but the data isn't very rich.
Also, please can we have some API tests in TestAPI. These are JUnit tests
in java. These should be have comprehensive coverage.
I also tried out execConstructTriples(), I noticed that the result is from
all graphs. When just triples are asked for, only those in the default
graph should be returned. Template.getTriples calls Quad.asTriple. But it
needs to deal with just the default graph, and c can't rely on
Quad.asTriple.
Andy
On 10/08/15 12:44, Qihong Lin wrote:
Hi,
I've enriched the syntax tests with short form and default graph
cases.
For execution tests, I add the test support in QueryTest for construct
quad with the scripts files and data files in TRIG (see
jena-arq/testing/ARQ/Construct/*.trig). I think construct quad should
be part of the construct of ARQ. So I add the execution tests in
jena-arq/testing/ARQ/Construct/manifest.ttl.
The fuseki part of construct quad has been implemented (not committed
yet). I'll submit the code as soon as the PR 89 [1] being merged.
Anything to be improved for PR 89 from your reviews?
regards,
Qihong
[1] https://github.com/apache/jena/pull/89
On Thu, Aug 6, 2015 at 8:55 PM, Andy Seaborne a...@apache.org wrote:
Hi - looking better - I see the syntax tests I provided.
Does that set of tests cover all cases of the new syntax?
Next, we need execution tests for ARQ.
There will need to be execute tests in jena-arq as well. These can be
Junit tests in java and do not need to be external script files.
This is separate from testing in Fuseki.
Do you have any other constraints on your time over the next 2 weeks?
Andy
On 06/08/15 04:18, Qihong Lin wrote:
Hi,
I've fixed the bugs related to syntax checks. Please see my latest
commits. Here're the output of the syntax files through qparse (you
can also run ExampleConstructQuads.java to get the results). Are
these with the expected behaviors, especially for
syntax-quad-construct-07.arq?
run-construct-quad-test: File: syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o } } WHERE {} Output of qparse
--file syntax-quad-construct-01.arq PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o .} } WHERE { } File:
syntax-quad-construct-02.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o } } WHERE { ?s ?p ?o } Output of
qparse --file syntax-quad-construct-02.arq PREFIX :
http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o .} } WHERE { ?s ?p ?o }
File: syntax-quad-construct-03.arq PREFIX : http://example/
CONSTRUCT { :s :p :o } WHERE {} Output of qparse --file
syntax-quad-construct-03.arq PREFIX : http://example/
CONSTRUCT { :s :p :o . } WHERE { } File:
syntax-quad-construct-04.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o } ?s ?p ?o } WHERE { GRAPH ?g { ?s ?p
?o } } Output of qparse --file syntax-quad-construct-04.arq
PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o .} ?s ?p ?o . } WHERE { GRAPH ?g { ?s
?p ?o } } File
[2/3] incubator-geode git commit: GEODE-77: Implement Authenticator interface in class GMSAuthenticator with unit tests.
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/0a70d514/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java
index cb4f9c9..76d9d71 100755
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java
@@ -1,1178 +1,1178 @@
-package com.gemstone.gemfire.distributed.internal.membership.gms.membership;
-
-import static
com.gemstone.gemfire.distributed.internal.DistributionManager.LOCATOR_DM_TYPE;
-import static
com.gemstone.gemfire.internal.DataSerializableFixedID.INSTALL_VIEW_MESSAGE;
-import static
com.gemstone.gemfire.internal.DataSerializableFixedID.JOIN_REQUEST;
-import static
com.gemstone.gemfire.internal.DataSerializableFixedID.JOIN_RESPONSE;
-import static
com.gemstone.gemfire.internal.DataSerializableFixedID.LEAVE_REQUEST_MESSAGE;
-import static
com.gemstone.gemfire.internal.DataSerializableFixedID.REMOVE_MEMBER_MESSAGE;
-import static
com.gemstone.gemfire.internal.DataSerializableFixedID.VIEW_ACK_MESSAGE;
-
-import java.io.IOException;
-import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashSet;
-import java.util.Iterator;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Set;
-import java.util.concurrent.Callable;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ThreadFactory;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.locks.ReadWriteLock;
-import java.util.concurrent.locks.ReentrantReadWriteLock;
-
-import org.apache.logging.log4j.Logger;
-
-import com.gemstone.gemfire.SystemConnectException;
-import com.gemstone.gemfire.distributed.internal.DistributionConfig;
-import com.gemstone.gemfire.distributed.internal.DistributionManager;
-import com.gemstone.gemfire.distributed.internal.DistributionMessage;
-import
com.gemstone.gemfire.distributed.internal.membership.InternalDistributedMember;
-import com.gemstone.gemfire.distributed.internal.membership.NetView;
-import com.gemstone.gemfire.distributed.internal.membership.gms.GMSMember;
-import com.gemstone.gemfire.distributed.internal.membership.gms.GMSUtil;
-import com.gemstone.gemfire.distributed.internal.membership.gms.Services;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.interfaces.JoinLeave;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.interfaces.MessageHandler;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.locator.FindCoordinatorRequest;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.locator.FindCoordinatorResponse;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.messages.InstallViewMessage;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.messages.JoinRequestMessage;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.messages.JoinResponseMessage;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.messages.LeaveRequestMessage;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.messages.RemoveMemberMessage;
-import
com.gemstone.gemfire.distributed.internal.membership.gms.messages.ViewAckMessage;
-import com.gemstone.gemfire.distributed.internal.tcpserver.TcpClient;
-import com.gemstone.gemfire.internal.Version;
-import com.gemstone.gemfire.internal.i18n.LocalizedStrings;
-import com.gemstone.gemfire.security.AuthenticationFailedException;
-
-/**
- * GMSJoinLeave handles membership communication with other processes in the
- * distributed system. It replaces the JGroups channel membership services
- * that Geode formerly used for this purpose.
- *
- */
-public class GMSJoinLeave implements JoinLeave, MessageHandler {
-
- /** number of times to try joining before giving up */
- private static final int JOIN_ATTEMPTS =
Integer.getInteger(gemfire.join-attempts, 4);
-
- /** amount of time to sleep before trying to join after a failed attempt */
- private static final int JOIN_RETRY_SLEEP =
Integer.getInteger(gemfire.join-retry-sleep, 1000);
-
- /** amount of time to wait for a view to be acked by all members before
performing suspect processing on non-responders */
- private static final int VIEW_INSTALLATION_TIMEOUT =
Integer.getInteger(gemfire.view-ack-timeout, 12500);
-
-
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi Andy, I can not find the branch of JENA-491-construct-quads in Jena git repository: https://github.com/apache/jena/branches regards, Qihong On Tue, Aug 11, 2015 at 4:12 PM, Andy Seaborne a...@apache.org wrote: On 11/08/15 03:35, Ying Jiang wrote: Hi Qihong, You may have received the reminder email from Google that the final evaluation of the GSoC project is approaching. The soft deadline is 17 August, with the hard one on 21 August. It means there're only 2 weeks left for your remaining work. The tests of arq need improving, as Andy points out. You can also deliver the Fuseki part in the other branch for review in the mean time. Meanwhile, the documentation can be composed in jena web page (wiki). In order to save time, I suggest the 3 parts of the work go on simultaneously: 1) jena/arq: in master branch (to be merged first) 2) jena/fuseki: in JENA-491 branch 3) doc: in jena web page (wiki) Best regards, Ying Jiang I have created a branch in the Jena git repository 'JENA-491-construct-quads' for doing this. Andy
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi Andy,
I've got the idea of projection. The bug has been fixed. I'll deliver
more tests.
Thanks!
Qihong
On Tue, Aug 11, 2015 at 4:52 PM, Andy Seaborne a...@apache.org wrote:
On 11/08/15 09:11, Qihong Lin wrote:
hi Andy,
I'll further enrich the test data and add JUnit tests in TestAPI for
better coverage.
I'm thinking of the problem you mentioned. In fact, there're 2 types
of query strings:
a) Query String without Construct Quad, e.g. CONSTRUCT { ?s ?p ?o }
WHERE { ?s ?p ?o };
b) Query String with Construct Quad, e.g. CONSTRUCT { ?s ?p ?o GRAPH
:g1 { :s1 ?p :o1 } } WHERE { ?s ?p ?o }
Also, the users can choose to execute 2 kinds of query methods:
1) IteratorTriple execTriples()
2) IteratorQuad execQuads()
These make 4 cases:
a)+1): No problem, original ARQ construct query
and execContructQuads returns quads with the default graph set.
and the execContructDataset contains a dataset with triples in the default
graph.
b)+2): No problem, new construct quad query
Yes
a)+2): The result quads are in the default graph? Or just throw
exception saying the query string not working with the query method?
Quads, with graph field being for the default graph.
b)+1)(you mentioned): The result triples contain only the ?s ?p ?o
part, ignoring construct GRAPH :g1 { :s1 ?p :o1 }? Or just throw
exception saying the query string not working with the query method?
Could you please tell me your opnions about the last 2 cases?
The effect should be triples in the default graph (?s ?p ?o). it's a
projection. The same happens if you parse, say TriG, into a model : just
the default graph is seen.
There is a general abstraction here (CONSTRUCT template to dataset). It's a
matter of how to handle the two API forms that deal with a restricted form
(triples/model). Seeing the default graph in each case is the way to deal
with it.
Andy
regards,
Qihong
On Mon, Aug 10, 2015 at 9:58 PM, Andy Seaborne a...@apache.org wrote:
I've managed to integrate the latest updates.
The scripted ones look, at first glance OK, but the data isn't very rich.
Also, please can we have some API tests in TestAPI. These are JUnit
tests
in java. These should be have comprehensive coverage.
I also tried out execConstructTriples(), I noticed that the result is
from
all graphs. When just triples are asked for, only those in the default
graph should be returned. Template.getTriples calls Quad.asTriple. But
it
needs to deal with just the default graph, and c can't rely on
Quad.asTriple.
Andy
On 10/08/15 12:44, Qihong Lin wrote:
Hi,
I've enriched the syntax tests with short form and default graph
cases.
For execution tests, I add the test support in QueryTest for construct
quad with the scripts files and data files in TRIG (see
jena-arq/testing/ARQ/Construct/*.trig). I think construct quad should
be part of the construct of ARQ. So I add the execution tests in
jena-arq/testing/ARQ/Construct/manifest.ttl.
The fuseki part of construct quad has been implemented (not committed
yet). I'll submit the code as soon as the PR 89 [1] being merged.
Anything to be improved for PR 89 from your reviews?
regards,
Qihong
[1] https://github.com/apache/jena/pull/89
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
hi Andy,
I'll further enrich the test data and add JUnit tests in TestAPI for
better coverage.
I'm thinking of the problem you mentioned. In fact, there're 2 types
of query strings:
a) Query String without Construct Quad, e.g. CONSTRUCT { ?s ?p ?o }
WHERE { ?s ?p ?o };
b) Query String with Construct Quad, e.g. CONSTRUCT { ?s ?p ?o GRAPH
:g1 { :s1 ?p :o1 } } WHERE { ?s ?p ?o }
Also, the users can choose to execute 2 kinds of query methods:
1) IteratorTriple execTriples()
2) IteratorQuad execQuads()
These make 4 cases:
a)+1): No problem, original ARQ construct query
b)+2): No problem, new construct quad query
a)+2): The result quads are in the default graph? Or just throw
exception saying the query string not working with the query method?
b)+1)(you mentioned): The result triples contain only the ?s ?p ?o
part, ignoring construct GRAPH :g1 { :s1 ?p :o1 }? Or just throw
exception saying the query string not working with the query method?
Could you please tell me your opnions about the last 2 cases?
regards,
Qihong
On Mon, Aug 10, 2015 at 9:58 PM, Andy Seaborne a...@apache.org wrote:
I've managed to integrate the latest updates.
The scripted ones look, at first glance OK, but the data isn't very rich.
Also, please can we have some API tests in TestAPI. These are JUnit tests
in java. These should be have comprehensive coverage.
I also tried out execConstructTriples(), I noticed that the result is from
all graphs. When just triples are asked for, only those in the default
graph should be returned. Template.getTriples calls Quad.asTriple. But it
needs to deal with just the default graph, and c can't rely on
Quad.asTriple.
Andy
On 10/08/15 12:44, Qihong Lin wrote:
Hi,
I've enriched the syntax tests with short form and default graph
cases.
For execution tests, I add the test support in QueryTest for construct
quad with the scripts files and data files in TRIG (see
jena-arq/testing/ARQ/Construct/*.trig). I think construct quad should
be part of the construct of ARQ. So I add the execution tests in
jena-arq/testing/ARQ/Construct/manifest.ttl.
The fuseki part of construct quad has been implemented (not committed
yet). I'll submit the code as soon as the PR 89 [1] being merged.
Anything to be improved for PR 89 from your reviews?
regards,
Qihong
[1] https://github.com/apache/jena/pull/89
On Thu, Aug 6, 2015 at 8:55 PM, Andy Seaborne a...@apache.org wrote:
Hi - looking better - I see the syntax tests I provided.
Does that set of tests cover all cases of the new syntax?
Next, we need execution tests for ARQ.
There will need to be execute tests in jena-arq as well. These can be
Junit tests in java and do not need to be external script files.
This is separate from testing in Fuseki.
Do you have any other constraints on your time over the next 2 weeks?
Andy
On 06/08/15 04:18, Qihong Lin wrote:
Hi,
I've fixed the bugs related to syntax checks. Please see my latest
commits. Here're the output of the syntax files through qparse (you
can also run ExampleConstructQuads.java to get the results). Are
these with the expected behaviors, especially for
syntax-quad-construct-07.arq?
run-construct-quad-test: File: syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o } } WHERE {} Output of qparse
--file syntax-quad-construct-01.arq PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o .} } WHERE { } File:
syntax-quad-construct-02.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o } } WHERE { ?s ?p ?o } Output of
qparse --file syntax-quad-construct-02.arq PREFIX :
http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o .} } WHERE { ?s ?p ?o }
File: syntax-quad-construct-03.arq PREFIX : http://example/
CONSTRUCT { :s :p :o } WHERE {} Output of qparse --file
syntax-quad-construct-03.arq PREFIX : http://example/
CONSTRUCT { :s :p :o . } WHERE { } File:
syntax-quad-construct-04.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o } ?s ?p ?o } WHERE { GRAPH ?g { ?s ?p
?o } } Output of qparse --file syntax-quad-construct-04.arq
PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o .} ?s ?p ?o . } WHERE { GRAPH ?g { ?s
?p ?o } } File: syntax-quad-construct-05.arq PREFIX :
http://example/
CONSTRUCT { ?s ?p ?o GRAPH ?g { :s :p :o } } WHERE { GRAPH ?g { ?s ?p
?o } } Output of qparse --file syntax-quad-construct-05.arq
PREFIX : http://example/
CONSTRUCT { ?s ?p ?o . GRAPH ?g { :s :p :o .} } WHERE { GRAPH ?g { ?s
?p ?o } } File: syntax-quad-construct-06.arq PREFIX :
http://example/
CONSTRUCT { GRAPH ?g { :s :p :o } ?s ?p ?o . ?s ?p ?o . GRAPH ?g { ?s
?p ?o } ?s ?p ?o . ?s ?p ?o GRAPH ?g { ?s ?p ?o } } WHERE { GRAPH ?g
{ ?s ?p ?o } } Output of qparse --file
syntax-quad-construct-06.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o .} ?s ?p
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi,
I've enriched the syntax tests with short form and default graph cases.
For execution tests, I add the test support in QueryTest for construct
quad with the scripts files and data files in TRIG (see
jena-arq/testing/ARQ/Construct/*.trig). I think construct quad should
be part of the construct of ARQ. So I add the execution tests in
jena-arq/testing/ARQ/Construct/manifest.ttl.
The fuseki part of construct quad has been implemented (not committed
yet). I'll submit the code as soon as the PR 89 [1] being merged.
Anything to be improved for PR 89 from your reviews?
regards,
Qihong
[1] https://github.com/apache/jena/pull/89
On Thu, Aug 6, 2015 at 8:55 PM, Andy Seaborne a...@apache.org wrote:
Hi - looking better - I see the syntax tests I provided.
Does that set of tests cover all cases of the new syntax?
Next, we need execution tests for ARQ.
There will need to be execute tests in jena-arq as well. These can be
Junit tests in java and do not need to be external script files.
This is separate from testing in Fuseki.
Do you have any other constraints on your time over the next 2 weeks?
Andy
On 06/08/15 04:18, Qihong Lin wrote:
Hi,
I've fixed the bugs related to syntax checks. Please see my latest
commits. Here're the output of the syntax files through qparse (you
can also run ExampleConstructQuads.java to get the results). Are
these with the expected behaviors, especially for
syntax-quad-construct-07.arq?
run-construct-quad-test: File: syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o } } WHERE {} Output of qparse
--file syntax-quad-construct-01.arq PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o .} } WHERE { } File:
syntax-quad-construct-02.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o } } WHERE { ?s ?p ?o } Output of
qparse --file syntax-quad-construct-02.arq PREFIX :
http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o .} } WHERE { ?s ?p ?o }
File: syntax-quad-construct-03.arq PREFIX : http://example/
CONSTRUCT { :s :p :o } WHERE {} Output of qparse --file
syntax-quad-construct-03.arq PREFIX : http://example/
CONSTRUCT { :s :p :o . } WHERE { } File:
syntax-quad-construct-04.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o } ?s ?p ?o } WHERE { GRAPH ?g { ?s ?p
?o } } Output of qparse --file syntax-quad-construct-04.arq
PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o .} ?s ?p ?o . } WHERE { GRAPH ?g { ?s
?p ?o } } File: syntax-quad-construct-05.arq PREFIX :
http://example/
CONSTRUCT { ?s ?p ?o GRAPH ?g { :s :p :o } } WHERE { GRAPH ?g { ?s ?p
?o } } Output of qparse --file syntax-quad-construct-05.arq
PREFIX : http://example/
CONSTRUCT { ?s ?p ?o . GRAPH ?g { :s :p :o .} } WHERE { GRAPH ?g { ?s
?p ?o } } File: syntax-quad-construct-06.arq PREFIX :
http://example/
CONSTRUCT { GRAPH ?g { :s :p :o } ?s ?p ?o . ?s ?p ?o . GRAPH ?g { ?s
?p ?o } ?s ?p ?o . ?s ?p ?o GRAPH ?g { ?s ?p ?o } } WHERE { GRAPH ?g
{ ?s ?p ?o } } Output of qparse --file
syntax-quad-construct-06.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o .} ?s ?p ?o . ?s ?p ?o . GRAPH ?g {
?s ?p ?o .} ?s ?p ?o . ?s ?p ?o . GRAPH ?g { ?s ?p ?o .} } WHERE {
GRAPH ?g { ?s ?p ?o } } File:
syntax-quad-construct-07.arq PREFIX : http://example/
CONSTRUCT { GRAPH urn:x-arq:DefaultGraphNode {:s :p :o .} } WHERE
{} Output of qparse --file syntax-quad-construct-07.arq PREFIX
: http://example/
CONSTRUCT { :s :p :o . } WHERE { } File:
syntax-quad-construct-08.arq PREFIX : http://example/
CONSTRUCT { GRAPH ?g { :s :p :o } GRAPH ?g1 { :s :p :o } } WHERE { }
Output of qparse --file syntax-quad-construct-08.arq PREFIX :
http://example/
CONSTRUCT { GRAPH ?g { :s :p :o .} GRAPH ?g1 { :s :p :o .} } WHERE {
}
On Sun, Aug 2, 2015 at 9:14 AM, Ying Jiang jpz6311...@gmail.com
wrote:
Hi Qihong,
I re-checked the code you commited. Most of the hashCode issues
for arq.qparse that Andy pointed out are due to the incorrect
serialization of the Template. The code can only deal with the
quads with the same graph node, for which it may not always be the
case in practcial terms. Please make sure each example syntax file
can pass the checking process of arq.qparse.
It's August now. You'd better hurry up and make the project
completed on time in the remaining weeks.
Best regards, Ying Jiang
On Thu, Jul 30, 2015 at 6:03 AM, Andy Seaborne a...@apache.org
wrote:
Output of * each examnple syntax file (the queries are nonsense
in practcial terms!) * running arq.qparse on the file * notes on
what seems to be going on
Andy
File: syntax-quad-construct-01.arq PREFIX :
http://example/
CONSTRUCT { GRAPH :g { :s :p :o } } WHERE {} Output of
qparse --file syntax-quad-construct-01.arq PREFIX :
http://example
[15/19] incubator-geode git commit: Fix GEODE-154: HDFS region with HeapLRU reports LowMemoryExceptions even when region size is less then the eviction threshold
Fix GEODE-154: HDFS region with HeapLRU reports LowMemoryExceptions
even when region size is less then the eviction threshold
- calling setUsageThreshold() on MemoryPoolMXBean resulted in
it reporting incorrect memory usage causing temporary spikes.
Since we already set CollectionThreshold at 1 byte and also have a
stat and heap poller thread, we now skip setting the usage threshold.
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/88ed3ed5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/88ed3ed5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/88ed3ed5
Branch: refs/heads/feature/GEODE-137
Commit: 88ed3ed59b96f8ce84327ed2bad132d4dc6e3a51
Parents: 389e2a8
Author: Namrata Thanvi ntha...@pivotal.io
Authored: Thu Aug 6 14:22:53 2015 +0530
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:48 2015 -0700
--
.../internal/cache/control/HeapMemoryMonitor.java | 13 +
1 file changed, 1 insertion(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/88ed3ed5/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java
index 8346245..fda337b 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/control/HeapMemoryMonitor.java
@@ -554,18 +554,7 @@ public void stopMonitoring() {
* Number of bytes of heap memory currently used.
*/
private void setUsageThresholdOnMXBean(final long bytesUsed) {
-if (testDisableMemoryUpdates) {
- return;
-}
-
-final MemoryPoolMXBean memoryPoolMXBean = getTenuredMemoryPoolMXBean();
-final MemoryThresholds saveThresholds = this.thresholds;
-
-if (bytesUsed saveThresholds.getEvictionThresholdBytes()) {
-
memoryPoolMXBean.setUsageThreshold(saveThresholds.getEvictionThresholdBytes());
-} else {
-
memoryPoolMXBean.setUsageThreshold(saveThresholds.getCriticalThresholdBytes());
-}
+ this method has been made a no-op to fix bug 49064
}
/**
[11/19] incubator-geode git commit: GEODE-172: Fixing an intermittent failure as reported by closing cache after each individual test.
GEODE-172: Fixing an intermittent failure as reported by closing cache after
each individual test.
Also temporarily taking care of an assertion failure that is reflected in
GEODE-110 and GEODE-141 that are still under investigation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/0a555b9e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/0a555b9e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/0a555b9e
Branch: refs/heads/feature/GEODE-137
Commit: 0a555b9e7e1e36fb3e9f651f3691ff50efee45d6
Parents: 3cef2d7
Author: Vivek Bhaskar vbhas...@pivotal.io
Authored: Thu Jul 30 17:14:57 2015 +0530
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:47 2015 -0700
--
.../gemfire/internal/cache/TXRegionState.java| 17 -
.../gemfire/disttx/DistTXDebugDUnitTest.java | 3 +++
.../disttx/DistributedTransactionDUnitTest.java | 19 +++
3 files changed, 22 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/0a555b9e/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXRegionState.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXRegionState.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXRegionState.java
index bb6ae5f..67cb8c5 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXRegionState.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXRegionState.java
@@ -596,11 +596,18 @@ public class TXRegionState {
String regionFullPath = this.getRegion().getFullPath();
int entryModsSize = this.entryMods.size();
int entryEventListSize = entryEventList.size();
-if (entryModsSize != entryEventListSize) {
- throw new UnsupportedOperationInTransactionException(
- LocalizedStrings.DISTTX_TX_EXPECTED.toLocalizedString(
- entry size of + entryModsSize + for region
- + regionFullPath, entryEventListSize));
+/*
+ * [DISTTX] TODO
+ * This assertion is not working for PutAll and RemoveAll operations
+ * and thus guarding within Debug flags. May be enabled at later stage.
+ */
+if (logger.isDebugEnabled()) {
+ if (entryModsSize != entryEventListSize) {
+throw new UnsupportedOperationInTransactionException(
+LocalizedStrings.DISTTX_TX_EXPECTED
+.toLocalizedString(entry size of + entryModsSize
++ for region + regionFullPath, entryEventListSize));
+ }
}
int index = 0;
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/0a555b9e/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistTXDebugDUnitTest.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistTXDebugDUnitTest.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistTXDebugDUnitTest.java
index ff0a506..668f1e6 100644
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistTXDebugDUnitTest.java
+++
b/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistTXDebugDUnitTest.java
@@ -62,6 +62,9 @@ public class DistTXDebugDUnitTest extends CacheTestCase {
}
});
InternalResourceManager.setResourceObserver(null);
+if (cache != null) {
+ cache.close();
+}
}
public static void createCacheInVm() {
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/0a555b9e/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistributedTransactionDUnitTest.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistributedTransactionDUnitTest.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistributedTransactionDUnitTest.java
index 2522e63..79d6f64 100644
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistributedTransactionDUnitTest.java
+++
b/gemfire-core/src/test/java/com/gemstone/gemfire/disttx/DistributedTransactionDUnitTest.java
@@ -72,13 +72,13 @@ public class DistributedTransactionDUnitTest extends
CacheTestCase {
}
});
-this.invokeInEveryVM(new SerializableCallable() {
- @Override
- public Object call() throws Exception {
-System.setProperty(gemfire.log-level, fine);
-return null;
- }
-});
+//this.invokeInEveryVM(new SerializableCallable() {
+// @Override
+// public Object call() throws Exception {
+//System.setProperty(gemfire.log-level, fine);
+//return null
[03/19] incubator-geode git commit: [GEODE-176] fixes the issue
[GEODE-176] fixes the issue
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/863c4ca3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/863c4ca3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/863c4ca3
Branch: refs/heads/feature/GEODE-137
Commit: 863c4ca3471b96344e26dda80e4d39c4f6a8da52
Parents: cd06fb9
Author: eshu e...@pivotal.io
Authored: Thu Jul 30 11:42:28 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:47 2015 -0700
--
.../gemfire/internal/offheap/SimpleMemoryAllocatorImpl.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/863c4ca3/gemfire-core/src/main/java/com/gemstone/gemfire/internal/offheap/SimpleMemoryAllocatorImpl.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/offheap/SimpleMemoryAllocatorImpl.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/offheap/SimpleMemoryAllocatorImpl.java
index edb24d1..29319e0 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/offheap/SimpleMemoryAllocatorImpl.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/offheap/SimpleMemoryAllocatorImpl.java
@@ -348,7 +348,7 @@ public final class SimpleMemoryAllocatorImpl implements
MemoryAllocator, MemoryI
SetBucketRegion brs = prs.getAllLocalBucketRegions();
if (brs != null) {
for (BucketRegion br : brs) {
- if (br != null) {
+ if (br != null !br.isDestroyed()) {
this.basicGetRegionLiveChunks(br, result);
}
[06/19] incubator-geode git commit: GEODE-162: remove race condititions and long sleeps from region expiration test methods in RegionTestCase
GEODE-162: remove race condititions and long sleeps from region expiration test
methods in RegionTestCase
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/cd06fb99
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/cd06fb99
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/cd06fb99
Branch: refs/heads/feature/GEODE-137
Commit: cd06fb9985c2afc29a53de83bcf9f117fae945ba
Parents: b081a9e
Author: Darrel Schneider dschnei...@pivotal.io
Authored: Thu Jul 30 16:40:14 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:47 2015 -0700
--
.../gemfire/internal/cache/LocalRegion.java | 7 +
.../gemfire/cache30/RegionTestCase.java | 132 ---
2 files changed, 60 insertions(+), 79 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/cd06fb99/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/LocalRegion.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/LocalRegion.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/LocalRegion.java
index 9ea26a7..495e992 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/LocalRegion.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/LocalRegion.java
@@ -8815,6 +8815,13 @@ public class LocalRegion extends AbstractRegion
}
return this.entryExpiryTasks.get(re);
}
+ /**
+ * Used by unit tests to get access to the RegionIdleExpiryTask
+ * of this region. Returns null if no task exists.
+ */
+ public RegionIdleExpiryTask getRegionIdleExpiryTask() {
+return this.regionIdleExpiryTask;
+ }
private void addExpiryTask(RegionEntry re, boolean ifAbsent)
{
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/cd06fb99/gemfire-core/src/test/java/com/gemstone/gemfire/cache30/RegionTestCase.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/cache30/RegionTestCase.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/cache30/RegionTestCase.java
index a20cfc5..69eeec0 100644
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/cache30/RegionTestCase.java
+++
b/gemfire-core/src/test/java/com/gemstone/gemfire/cache30/RegionTestCase.java
@@ -2085,7 +2085,7 @@ public abstract class RegionTestCase extends
CacheTestCase {
if (now = tilt)
break;
if (!region.isDestroyed()) {
-pause(100);
+pause(10);
continue;
}
if (now = tilt - SLOP) {
@@ -2107,7 +2107,7 @@ public abstract class RegionTestCase extends
CacheTestCase {
break;
Assert.assertTrue(System.currentTimeMillis() = tilt,
Region failed to destroy);
- pause(100);
+ pause(10);
}
}
@@ -2704,7 +2704,7 @@ public abstract class RegionTestCase extends
CacheTestCase {
vm0.invoke(new CacheSerializableRunnable(testRegionTtlInvalidate) {
public void run2() throws CacheException {
-final int timeout = 50; // ms
+final int timeout = 22; // ms
final Object key = KEY;
final Object value = VALUE;
@@ -2736,8 +2736,8 @@ public abstract class RegionTestCase extends
CacheTestCase {
finally {
ExpiryTask.permitExpiration();
}
-waitForInvalidate(entry, tilt);
-waitForInvalidate(region.getEntry(k2), tilt);
+waitForInvalidate(entry, tilt, 10);
+waitForInvalidate(region.getEntry(k2), tilt, 10);
}
});
}
@@ -2753,7 +2753,7 @@ public abstract class RegionTestCase extends
CacheTestCase {
return;
final String name = this.getUniqueName();
-final int timeout = 700; // ms
+final int timeout = 22; // ms
final Object key = KEY;
final Object value = VALUE;
@@ -3741,42 +3741,38 @@ public abstract class RegionTestCase extends
CacheTestCase {
throws CacheException, InterruptedException {
final String name = this.getUniqueName();
-final int timeout = 20; // ms
-final int hugeTimeout = Integer.MAX_VALUE;
-final ExpirationAttributes expire =
-new ExpirationAttributes(timeout, ExpirationAction.INVALIDATE);
-final ExpirationAttributes hugeExpire =
-new ExpirationAttributes(hugeTimeout, ExpirationAction.INVALIDATE);
+;
final Object key = KEY;
final Object value = VALUE;
AttributesFactory factory = new AttributesFactory(getRegionAttributes());
factory.setStatisticsEnabled(true);
RegionAttributes attrs = factory.create();
-Region region = null;
-long tilt
[13/19] incubator-geode git commit: GEODE-155: - Fixes intermittent failure in RegionWithHDFS*DUnitTest about incorrect number of hoplog files getting created. - Caused due to an earlier test not clea
GEODE-155:
- Fixes intermittent failure in RegionWithHDFS*DUnitTest about incorrect number
of hoplog files getting created.
- Caused due to an earlier test not cleaning up static variables in
ParallelGatewaySenderQueue.
- These are indeed cleaned up during GemFireCacheImpl.close() if there are any
GatewaySenders running in the system.
- But if a region with gateway-senders associated with it, is destroyed first
before a cache.close(), then the gateway senders are only stopped,
and removed from allGatewaySenders list. But the static variables are not
cleaned up.
- Later, during GemFireCacheImpl.close(), as the allGatewaySenders list is
empty, it skips cleaning the static variables.
- As a fix, invoking a static clean-up method of ParallelGatewaySenderQueue
explicitly during cache.close().
- Retained the non-static ParallelGatewaySenderQueue.cleanUp(), if required in
future development for performing instance-specific clean-up.
- Minor formatting changes in the unit tests.
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/cb735a20
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/cb735a20
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/cb735a20
Branch: refs/heads/feature/GEODE-137
Commit: cb735a2045f124ee7c5c6ba84dbcafe4383008d7
Parents: 71f00b5
Author: ashetkar ashet...@pivotal.io
Authored: Tue Aug 4 14:06:51 2015 +0530
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:48 2015 -0700
--
.../internal/cache/GemFireCacheImpl.java| 1 +
.../parallel/ParallelGatewaySenderQueue.java| 41 +---
.../internal/RegionWithHDFSBasicDUnitTest.java | 22 +--
.../RegionWithHDFSOffHeapBasicDUnitTest.java| 10 +++--
.../hdfs/internal/RegionWithHDFSTestBase.java | 6 +--
5 files changed, 49 insertions(+), 31 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/cb735a20/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/GemFireCacheImpl.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/GemFireCacheImpl.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/GemFireCacheImpl.java
index 5487000..f5be144 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/GemFireCacheImpl.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/GemFireCacheImpl.java
@@ -2018,6 +2018,7 @@ public class GemFireCacheImpl implements InternalCache,
ClientCache, HasCachePer
advisor.close();
}
}
+ ParallelGatewaySenderQueue.cleanUpStatics(null);
} catch (CancelException ce) {
}
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/cb735a20/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/parallel/ParallelGatewaySenderQueue.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/parallel/ParallelGatewaySenderQueue.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/parallel/ParallelGatewaySenderQueue.java
index f4f9528..9141905 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/parallel/ParallelGatewaySenderQueue.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/parallel/ParallelGatewaySenderQueue.java
@@ -633,15 +633,20 @@ public class ParallelGatewaySenderQueue implements
RegionQueue {
* Wait a while for existing tasks to terminate. If the existing tasks still
don't
* complete, cancel them by calling shutdownNow.
*/
- private void cleanupConflationThreadPool() {
+ private static void cleanupConflationThreadPool(AbstractGatewaySender
sender) {
conflationExecutor.shutdown();// Disable new tasks from being submitted
try {
if (!conflationExecutor.awaitTermination(1, TimeUnit.SECONDS)) {
conflationExecutor.shutdownNow(); // Cancel currently executing tasks
// Wait a while for tasks to respond to being cancelled
- if (!conflationExecutor.awaitTermination(1, TimeUnit.SECONDS))
-
logger.warn(LocalizedMessage.create(LocalizedStrings.ParallelGatewaySenderQueue_COULD_NOT_TERMINATE_CONFLATION_THREADPOOL,
this.sender));
+if (!conflationExecutor.awaitTermination(1, TimeUnit.SECONDS)) {
+ logger
+ .warn(LocalizedMessage
+ .create(
+
LocalizedStrings.ParallelGatewaySenderQueue_COULD_NOT_TERMINATE_CONFLATION_THREADPOOL,
+ (sender == null ? all : sender)));
+}
}
} catch (InterruptedException e
[1/2] incubator-geode git commit: GEODE-137: use local GemFire server to initialize ClientCache whenever possible.
Repository: incubator-geode
Updated Branches:
refs/heads/develop 7d4ae09fc - d0d48588c
GEODE-137: use local GemFire server to initialize ClientCache whenever possible.
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/d0d48588
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/d0d48588
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/d0d48588
Branch: refs/heads/develop
Commit: d0d48588c895cff7418090c50b57f9f4a740763c
Parents: 2e2a795
Author: Qihong Chen qc...@pivotal.io
Authored: Thu Aug 6 09:51:53 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:44:36 2015 -0700
--
.../pivotal/gemfire/spark/connector/internal/LocatorHelper.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/d0d48588/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/LocatorHelper.scala
--
diff --git
a/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/LocatorHelper.scala
b/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/LocatorHelper.scala
index a010c62..3a094cc 100644
---
a/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/LocatorHelper.scala
+++
b/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/LocatorHelper.scala
@@ -65,8 +65,8 @@ object LocatorHelper extends Logging {
* Pick up at most 3 preferred servers from all available servers based on
* host name and Spark executor id.
*
- * This method is used by DefaultGemFireConnection to create LocalCache.
Usually
- * one server is enough to initialize LocalCacheFactory, but this provides
two
+ * This method is used by DefaultGemFireConnection to create ClientCache.
Usually
+ * one server is enough to initialize ClientCacheFactory, but this provides
two
* backup servers in case of the 1st server can't be connected.
*
* @param servers all available servers in the form of (hostname, port) pairs
Git Push Summary
Repository: incubator-geode Updated Branches: refs/heads/feature/GEODE-137 [deleted] e596a4eec
[02/19] incubator-geode git commit: GEODE-168: remove Bug34011JUnitTest
GEODE-168: remove Bug34011JUnitTest
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/daeb077f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/daeb077f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/daeb077f
Branch: refs/heads/feature/GEODE-137
Commit: daeb077f099545346f2ac8080689b2fe23db4aec
Parents: 6f6075f
Author: Darrel Schneider dschnei...@pivotal.io
Authored: Wed Jul 29 14:47:42 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:46 2015 -0700
--
.../internal/cache/Bug34011JUnitTest.java | 199 ---
1 file changed, 199 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/daeb077f/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/Bug34011JUnitTest.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/Bug34011JUnitTest.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/Bug34011JUnitTest.java
deleted file mode 100644
index b673a56..000
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/Bug34011JUnitTest.java
+++ /dev/null
@@ -1,199 +0,0 @@
-/*=
- * Copyright (c) 2010-2014 Pivotal Software, Inc. All Rights Reserved.
- * This product is protected by U.S. and international copyright
- * and intellectual property laws. Pivotal products are covered by
- * one or more patents listed at http://www.pivotal.io/patents.
- *=
- */
-package com.gemstone.gemfire.internal.cache;
-
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
-import java.util.Arrays;
-
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.experimental.categories.Category;
-
-import static org.junit.Assert.*;
-
-import com.gemstone.gemfire.cache.Region;
-import com.gemstone.gemfire.test.junit.categories.IntegrationTest;
-
-/**
- * To verify the bug no. 34011 no longer exists:
- * Disk region perf test for Persist only with Async writes.
- *
- * The test verifies that puts per second perfomance if bytes threshold being
exceeded before time interval is not more than a factor
- * of 2 (the reverse is also tested, time interval causing flush's perfomance
should not be better than byte-threshold exceeding by
- * a factor of 2)
- *
- * Note : This test can fail due to external factors such as filesystem
becoming slow or CPU being overloaded
- * during one run and fast during the second run.
- *
- * @author Vikram Jadhav
- */
-@Category(IntegrationTest.class)
-public class Bug34011JUnitTest extends DiskRegionTestingBase
-{
- String stats1 = null;
- String stats2 = null;
- DiskRegionProperties diskProps1 = new DiskRegionProperties();
- DiskRegionProperties diskProps2 = new DiskRegionProperties();
- Region region1= null;
- Region region2= null;
- public float opPerSec1= 0l;
- public float opPerSec2= 0l;
-
- @Before
- public void setUp() throws Exception
- {
-super.setUp();
-diskProps1.setDiskDirs(dirs);
-diskProps2.setDiskDirs(dirs);
-DiskStoreImpl.SET_IGNORE_PREALLOCATE = true;
- }
-
- @After
- public void tearDown() throws Exception
- {
-super.tearDown();
-DiskStoreImpl.SET_IGNORE_PREALLOCATE = false;
- }
-
- private static int ENTRY_SIZE = 2;
-
- private static int OP_COUNT = 10; // 10;
-
-
- /**
- * First, the time interval is set to a low value such that time-interval
always elapses before bytes threshold is reached.
- * Then the bytes-threshold is set in such a way that byte threshold occurs
before time-interval. The perfomance
- * of the first one should not be more than a factor of two as compared to
the perfomance of the second scenario. The
- * reverse also hold true
- *
- */
-
- @Test
- public void testpersistASync()
- {
-
-// test-persistASync-ByteThreshold
-try {
-
- diskProps1.setTimeInterval(10);
- diskProps1.setBytesThreshold(Integer.MAX_VALUE); // now a queue size
- diskProps1.setRegionName(region1);
- region1 = DiskRegionHelperFactory.getAsyncPersistOnlyRegion(cache,
diskProps1);
-
-}
-catch (Exception e) {
- if(logWriter.fineEnabled()){
-e.printStackTrace();
- }
- fail(failed : test-persistASync-ByteThreshold.Exception=+e);
-}
-//Perf test for 1kb writes
-populateData1();
-if(logWriter.infoEnabled()){
-logWriter.info(testpersistASyncByteThreshold:: Stats for 1 kb writes :
-+ stats1);
-}
- // close region1
-region1.close();
-
-
-try
[01/19] incubator-geode git commit: GEODE-169: testEventsExpiryBug should expect Unexpected IOException.
Repository: incubator-geode
Updated Branches:
refs/heads/feature/GEODE-137 6f6075f27 - e596a4eec
GEODE-169: testEventsExpiryBug should expect Unexpected IOException.
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/3cef2d71
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/3cef2d71
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/3cef2d71
Branch: refs/heads/feature/GEODE-137
Commit: 3cef2d7103851b4f840a533c808ca80692e35f7a
Parents: daeb077
Author: Darrel Schneider dschnei...@pivotal.io
Authored: Wed Jul 29 15:31:08 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:46 2015 -0700
--
.../gemfire/internal/cache/ha/Bug36853EventsExpiryDUnitTest.java| 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/3cef2d71/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ha/Bug36853EventsExpiryDUnitTest.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ha/Bug36853EventsExpiryDUnitTest.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ha/Bug36853EventsExpiryDUnitTest.java
index 00336e9..b22ac27 100755
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ha/Bug36853EventsExpiryDUnitTest.java
+++
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ha/Bug36853EventsExpiryDUnitTest.java
@@ -234,6 +234,7 @@ public class Bug36853EventsExpiryDUnitTest extends
CacheTestCase
*/
public void testEventsExpiryBug() throws Exception
{
+addExpectedException(Unexpected IOException);
addExpectedException(Connection reset);
server.invoke(Bug36853EventsExpiryDUnitTest.class, generateEvents);
client.invoke(Bug36853EventsExpiryDUnitTest.class,
[18/19] incubator-geode git commit: Fix GEODE-186 by removing sleeps in tests
Fix GEODE-186 by removing sleeps in tests
The old test scheduled tx suspension to timeout after 1 minute.
So the test always run for at least 1 minute.
A test hook now exists that allows the test to specify
a different time unit (default is still minutes) for
tx suspension expiration.
The sleeps in a bunch of other tests were not needed
since the tx operation is synchronous. So those sleeps
have simply been removed.
A couple of sleeps in clients waiting for something to
arrive that was done on a server have been converted to
a wait since server to client distribution is async.
Reviewed by Swapnil.
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/389e2a8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/389e2a8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/389e2a8c
Branch: refs/heads/feature/GEODE-137
Commit: 389e2a8cac1f2b35aa3ebfd10e8c49ae993b0dc4
Parents: e4c77c6
Author: Darrel Schneider dschnei...@pivotal.io
Authored: Tue Aug 4 11:43:39 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:48 2015 -0700
--
.../gemfire/internal/cache/TXManagerImpl.java | 11 +--
.../cache/ClientServerTransactionDUnitTest.java | 32 ++--
.../cache/RemoteTransactionDUnitTest.java | 30 --
3 files changed, 45 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/389e2a8c/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXManagerImpl.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXManagerImpl.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXManagerImpl.java
index 88714b0..dde3793 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXManagerImpl.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/cache/TXManagerImpl.java
@@ -1117,6 +1117,10 @@ public final class TXManagerImpl implements
CacheTransactionManager,
private ConcurrentMapTransactionId, TXStateProxy suspendedTXs = new
ConcurrentHashMapTransactionId, TXStateProxy();
public TransactionId suspend() {
+return suspend(TimeUnit.MINUTES);
+ }
+
+ TransactionId suspend(TimeUnit expiryTimeUnit) {
TXStateProxy result = getTXState();
if (result != null) {
TransactionId txId = result.getTransactionId();
@@ -1137,7 +1141,7 @@ public final class TXManagerImpl implements
CacheTransactionManager,
LockSupport.unpark(waitingThread);
}
}
- scheduleExpiry(txId);
+ scheduleExpiry(txId, expiryTimeUnit);
return txId;
}
return null;
@@ -1266,8 +1270,9 @@ public final class TXManagerImpl implements
CacheTransactionManager,
/**
* schedules the transaction to expire after {@link #suspendedTXTimeout}
* @param txId
+ * @param expiryTimeUnit the time unit to use when scheduling the expiration
*/
- private void scheduleExpiry(TransactionId txId) {
+ private void scheduleExpiry(TransactionId txId, TimeUnit expiryTimeUnit) {
final GemFireCacheImpl cache = (GemFireCacheImpl) this.cache;
if (suspendedTXTimeout 0) {
if (logger.isDebugEnabled()) {
@@ -1279,7 +1284,7 @@ public final class TXManagerImpl implements
CacheTransactionManager,
if (logger.isDebugEnabled()) {
logger.debug(TX: scheduling transaction: {} to expire after:{}, txId,
suspendedTXTimeout);
}
-cache.getCCPTimer().schedule(task, suspendedTXTimeout*60*1000);
+cache.getCCPTimer().schedule(task,
TimeUnit.MILLISECONDS.convert(suspendedTXTimeout, expiryTimeUnit));
this.expiryTasks.put(txId, task);
}
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/389e2a8c/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ClientServerTransactionDUnitTest.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ClientServerTransactionDUnitTest.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ClientServerTransactionDUnitTest.java
index 3ce8cec..d80f6bb 100644
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ClientServerTransactionDUnitTest.java
+++
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/ClientServerTransactionDUnitTest.java
@@ -16,6 +16,7 @@ import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
import javax.naming.Context;
import javax.transaction.UserTransaction;
@@ -78,6 +79,7 @@ import
[09/19] incubator-geode git commit: GEODE-181: Add new unit and integration tests for FastLogger
GEODE-181: Add new unit and integration tests for FastLogger
Add new TestSuites for logging tests. Extract default config testing to
FastLoggerWithDefaultConfigJUnitTest. Move FastLoggerJUnitTest integration
tests to FastLoggerIntegrationJUnitTest. Create new unit tests in
FastLoggerJUnitTest. Update tests to full JUnit 4 syntax and use Rules
(GEODE-89).
Add mockito dependency to build for unit testing.
Completes: GEODE-89, GEODE-181, GEODE-10
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/946f6b61
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/946f6b61
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/946f6b61
Branch: refs/heads/feature/GEODE-137
Commit: 946f6b61ce66fec7b8f960cf91add816f17e551f
Parents: a9da142
Author: Kirk Lund kl...@pivotal.io
Authored: Tue Aug 4 11:39:44 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:47 2015 -0700
--
build.gradle| 1 +
.../gemfire/internal/logging/LogService.java| 53 +-
.../internal/logging/log4j/Configurator.java| 65 ++-
.../internal/logging/log4j/FastLogger.java | 48 +-
.../DistributedSystemLogFileJUnitTest.java | 20 +-
.../logging/LoggingIntegrationTestSuite.java| 14 +
.../internal/logging/LoggingUnitTestSuite.java | 12 +
.../log4j/FastLoggerIntegrationJUnitTest.java | 557 +++
.../logging/log4j/FastLoggerJUnitTest.java | 274 -
.../FastLoggerWithDefaultConfigJUnitTest.java | 74 +++
.../log4j/Log4jIntegrationTestSuite.java| 12 +
.../logging/log4j/Log4jUnitTestSuite.java | 16 +
12 files changed, 914 insertions(+), 232 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/946f6b61/build.gradle
--
diff --git a/build.gradle b/build.gradle
index c82e82a..42b4a14 100755
--- a/build.gradle
+++ b/build.gradle
@@ -278,6 +278,7 @@ subprojects {
testCompile 'com.github.stefanbirkner:system-rules:1.9.0'
testCompile 'edu.umd.cs.mtc:multithreadedtc:1.01'
testCompile 'junit:junit:4.12'
+testCompile 'org.mockito:mockito-core:1.10.19'
testCompile 'org.hamcrest:hamcrest-all:1.3'
testCompile 'org.jmock:jmock:2.8.1'
testCompile 'org.jmock:jmock-junit4:2.8.1'
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/946f6b61/gemfire-core/src/main/java/com/gemstone/gemfire/internal/logging/LogService.java
--
diff --git
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/logging/LogService.java
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/logging/LogService.java
index 6298cf6..a4a399d 100644
---
a/gemfire-core/src/main/java/com/gemstone/gemfire/internal/logging/LogService.java
+++
b/gemfire-core/src/main/java/com/gemstone/gemfire/internal/logging/LogService.java
@@ -12,6 +12,7 @@ import java.beans.PropertyChangeListener;
import java.io.File;
import java.net.URISyntaxException;
import java.net.URL;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicReference;
@@ -21,9 +22,11 @@ import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.core.Appender;
import org.apache.logging.log4j.core.Filter.Result;
import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.config.AppenderRef;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.ConfigurationFactory;
import org.apache.logging.log4j.core.config.LoggerConfig;
+import org.apache.logging.log4j.core.filter.AbstractFilterable;
import org.apache.logging.log4j.core.lookup.Interpolator;
import org.apache.logging.log4j.core.lookup.StrSubstitutor;
import org.apache.logging.log4j.status.StatusLogger;
@@ -78,10 +81,9 @@ public class LogService extends LogManager {
private static void init() {
setLog4jConfigFileProperty();
LoggerContext context = ((org.apache.logging.log4j.core.Logger)
LogManager.getLogger(BASE_LOGGER_NAME,
GemFireParameterizedMessageFactory.INSTANCE)).getContext();
-context.reconfigure();
context.removePropertyChangeListener(propertyChangeListener);
context.addPropertyChangeListener(propertyChangeListener);
-setFastLoggerDebugAvailableFlag();
+context.reconfigure(); // propertyChangeListener invokes
configureFastLoggerDelegating
configureLoggers(false, false);
}
@@ -117,15 +119,17 @@ public class LogService extends LogManager {
/**
* Check to see if the user has specified a Log4j configuration file. If
not, attempt
* to find a GemFire Log4j configuration file in various locations
[10/19] incubator-geode git commit: GEODE-175: change test to use a wait criteria instead of pause and use millis instead of seconds
GEODE-175: change test to use a wait criteria instead of pause and use millis
instead of seconds
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/a9da1422
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/a9da1422
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/a9da1422
Branch: refs/heads/feature/GEODE-137
Commit: a9da1422d11d8944a210bae1294c9facd2265a80
Parents: 32186b9
Author: Darrel Schneider dschnei...@pivotal.io
Authored: Tue Aug 4 10:42:50 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Thu Aug 6 10:07:47 2015 -0700
--
.../cache/RemoteTransactionDUnitTest.java | 42 +---
1 file changed, 27 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/a9da1422/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/RemoteTransactionDUnitTest.java
--
diff --git
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/RemoteTransactionDUnitTest.java
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/RemoteTransactionDUnitTest.java
index 7c6d133..0daaafb 100644
---
a/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/RemoteTransactionDUnitTest.java
+++
b/gemfire-core/src/test/java/com/gemstone/gemfire/internal/cache/RemoteTransactionDUnitTest.java
@@ -4006,10 +4006,15 @@ protected static class ClientListener extends
CacheListenerAdapter {
vm1.invoke(new SerializableCallable() {
@Override
public Object call() throws Exception {
+System.setProperty(LocalRegion.EXPIRY_MS_PROPERTY, true);
+try {
RegionFactoryString, String rf = getCache().createRegionFactory();
rf.setEntryTimeToLive(new ExpirationAttributes(1,
ExpirationAction.LOCAL_DESTROY));
rf.setScope(Scope.DISTRIBUTED_ACK);
rf.create(regionName);
+} finally {
+ System.getProperties().remove(LocalRegion.EXPIRY_MS_PROPERTY);
+}
return null;
}
});
@@ -4027,20 +4032,6 @@ protected static class ClientListener extends
CacheListenerAdapter {
@Override
public Object call() throws Exception {
final RegionString, String r = getCache().getRegion(regionName);
-r.put(key, value);
-r.put(nonTXkey, nonTXvalue);
-getCache().getCacheTransactionManager().begin();
-r.put(key, newvalue);
-// wait for entry to expire
-DistributedTestCase.pause(5000);
-TransactionId tx = getCache().getCacheTransactionManager().suspend();
-// A remote tx will allow expiration to happen on the side that
-// is not hosting the tx. But it will not allow an expiration
-// initiated on the hosting jvm.
-assertFalse(r.containsKey(key));
-assertFalse(r.containsKey(nonTXkey));
-getCache().getCacheTransactionManager().resume(tx);
-getCache().getCacheTransactionManager().commit();
WaitCriterion wc2 = new WaitCriterion() {
@Override
public boolean done() {
@@ -4049,9 +4040,30 @@ protected static class ClientListener extends
CacheListenerAdapter {
@Override
public String description() {
-return did not expire;
+return did not expire containsKey(key)= + r.containsKey(key) +
r.containsKey(nonTXKey)= + r.containsKey(nonTXKey);
}
};
+ExpiryTask.suspendExpiration();
+Region.Entry entry = null;
+long tilt;
+try {
+ r.put(key, value);
+ r.put(nonTXkey, nonTXvalue);
+ getCache().getCacheTransactionManager().begin();
+ r.put(key, newvalue);
+}
+finally {
+ ExpiryTask.permitExpiration();
+}
+TransactionId tx = getCache().getCacheTransactionManager().suspend();
+// A remote tx will allow expiration to happen on the side that
+// is not hosting the tx. But it will not allow an expiration
+// initiated on the hosting jvm.
+// tx is hosted in vm2 so expiration can happen in vm1.
+DistributedTestCase.waitForCriterion(wc2, 3, 5, true);
+getCache().getCacheTransactionManager().resume(tx);
+assertTrue(r.containsKey(key));
+getCache().getCacheTransactionManager().commit();
DistributedTestCase.waitForCriterion(wc2, 3, 5, true);
return null;
}
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi,
I've fixed the bugs related to syntax checks. Please see my latest
commits. Here're the output of the syntax files through qparse (you
can also run ExampleConstructQuads.java to get the results). Are
these with the expected behaviors, especially for
syntax-quad-construct-07.arq?
run-construct-quad-test:
File: syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o } } WHERE {}
Output of qparse --file syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT
{
GRAPH :g
{ :s :p :o .}
}
WHERE
{ }
File: syntax-quad-construct-02.arq
PREFIX : http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o } } WHERE { ?s ?p ?o }
Output of qparse --file syntax-quad-construct-02.arq
PREFIX : http://example/
CONSTRUCT
{
GRAPH ?g
{ ?s ?p ?o .}
}
WHERE
{ ?s ?p ?o }
File: syntax-quad-construct-03.arq
PREFIX : http://example/
CONSTRUCT { :s :p :o } WHERE {}
Output of qparse --file syntax-quad-construct-03.arq
PREFIX : http://example/
CONSTRUCT
{
:s :p :o .
}
WHERE
{ }
File: syntax-quad-construct-04.arq
PREFIX : http://example/
CONSTRUCT {
GRAPH ?g { :s :p :o }
?s ?p ?o
}
WHERE
{ GRAPH ?g { ?s ?p ?o } }
Output of qparse --file syntax-quad-construct-04.arq
PREFIX : http://example/
CONSTRUCT
{
GRAPH ?g
{ :s :p :o .}
?s ?p ?o .
}
WHERE
{ GRAPH ?g
{ ?s ?p ?o }
}
File: syntax-quad-construct-05.arq
PREFIX : http://example/
CONSTRUCT {
?s ?p ?o
GRAPH ?g { :s :p :o }
}
WHERE
{ GRAPH ?g { ?s ?p ?o } }
Output of qparse --file syntax-quad-construct-05.arq
PREFIX : http://example/
CONSTRUCT
{
?s ?p ?o .
GRAPH ?g
{ :s :p :o .}
}
WHERE
{ GRAPH ?g
{ ?s ?p ?o }
}
File: syntax-quad-construct-06.arq
PREFIX : http://example/
CONSTRUCT {
GRAPH ?g { :s :p :o }
?s ?p ?o .
?s ?p ?o .
GRAPH ?g { ?s ?p ?o }
?s ?p ?o .
?s ?p ?o
GRAPH ?g { ?s ?p ?o }
}
WHERE
{ GRAPH ?g { ?s ?p ?o } }
Output of qparse --file syntax-quad-construct-06.arq
PREFIX : http://example/
CONSTRUCT
{
GRAPH ?g
{ :s :p :o .}
?s ?p ?o .
?s ?p ?o .
GRAPH ?g
{ ?s ?p ?o .}
?s ?p ?o .
?s ?p ?o .
GRAPH ?g
{ ?s ?p ?o .}
}
WHERE
{ GRAPH ?g
{ ?s ?p ?o }
}
File: syntax-quad-construct-07.arq
PREFIX : http://example/
CONSTRUCT {
GRAPH urn:x-arq:DefaultGraphNode {:s :p :o .}
}
WHERE {}
Output of qparse --file syntax-quad-construct-07.arq
PREFIX : http://example/
CONSTRUCT
{
:s :p :o .
}
WHERE
{ }
File: syntax-quad-construct-08.arq
PREFIX : http://example/
CONSTRUCT {
GRAPH ?g { :s :p :o }
GRAPH ?g1 { :s :p :o }
}
WHERE
{ }
Output of qparse --file syntax-quad-construct-08.arq
PREFIX : http://example/
CONSTRUCT
{
GRAPH ?g
{ :s :p :o .}
GRAPH ?g1
{ :s :p :o .}
}
WHERE
{ }
On Sun, Aug 2, 2015 at 9:14 AM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
I re-checked the code you commited. Most of the hashCode issues for
arq.qparse that Andy pointed out are due to the incorrect
serialization of the Template. The code can only deal with the quads
with the same graph node, for which it may not always be the case in
practcial terms. Please make sure each example syntax file can pass
the checking process of arq.qparse.
It's August now. You'd better hurry up and make the project completed
on time in the remaining weeks.
Best regards,
Ying Jiang
On Thu, Jul 30, 2015 at 6:03 AM, Andy Seaborne a...@apache.org wrote:
Output of
* each examnple syntax file (the queries are nonsense in practcial terms!)
* running arq.qparse on the file
* notes on what seems to be going on
Andy
File: syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT { GRAPH :g { :s :p :o } } WHERE {}
Output of qparse --file syntax-quad-construct-01.arq
PREFIX : http://example/
CONSTRUCT
{ GRAPH http://example/g {:s :p :o .}
}
WHERE
{ }
Issue:
Minor: formatting of http://example/g should be :g
File: syntax-quad-construct-02.arq
PREFIX : http://example/
CONSTRUCT { GRAPH ?g { ?s ?p ?o } } WHERE { ?s ?p ?o }
Output of qparse --file syntax-quad-construct-02.arq
PREFIX : http://example/
CONSTRUCT
{ GRAPH ?g {?s ?p ?o .}
}
WHERE
{ ?s ?p ?o }
OK
File: syntax-quad-construct-03.arq
PREFIX : http://example/
CONSTRUCT { :s :p :o } WHERE {}
Output of qparse --file syntax-quad-construct-03.arq
PREFIX : http://example/
CONSTRUCT
{ GRAPH urn:x-arq:DefaultGraphNode {:s :p :o .}
}
WHERE
{ }
Issue:
Major: The output is contains GRAPH urn:x-arq:DefaultGraphNode
Note: urn:x-arq:DefaultGraphNode is for internal use only.
For syntax, that is a named graph just like http://example/g
File: syntax-quad
Re: Fwd: Final Deliveries of GSoC Project (JENA-491)
Hi, I've fixed the bugs of arq.aparse and , with some minor improvements for Java 7/8 you mentioned. Please check the PR 89. I hope it can be merged into jena master ASAP, so that I can go on with merging the code for fuseki support. regard, Qihong On Wed, Jul 29, 2015 at 11:54 AM, Ying Jiang jpz6311...@gmail.com wrote: Hi Qihong, You're requried to keep us posted of your work progress regularly. Last week, there were some comments for your code ( PR 89 ) from the community. Have you read them and refined your code accordingly? Any questions? Best regards, Ying Jiang On Tue, Jul 21, 2015 at 4:35 PM, Andy Seaborne a...@apache.org wrote: On 21/07/15 17:46, Ying Jiang wrote: Hi Andy, Does Qihong need to be an apache committer in order to have the commit access? Can you grant her the access right? Best regards, Ying Jiang When you press Improve this page it says please use the username anonymous and leave the password blank if prompted for credentials Before documentation makes sense, there needs to code updates. This is most important. http://mail-archives.apache.org/mod_mbox/jena-dev/201507.mbox/%3C55ABD740.9010101%40apache.org%3E 1. Code and grammar changes 2. Tests 3. Pull requests 4. Documentation http://mail-archives.apache.org/mod_mbox/jena-dev/201507.mbox/%3C559EF342.3010806%40apache.org%3E Andy -- Forwarded message -- From: Qihong Lin confidence@gmail.com Date: Tue, Jul 21, 2015 at 9:25 AM Subject: Fwd: Final Deliveries of GSoC Project (JENA-491) To: Ying Jiang jpz6311...@gmail.com Hi, Dr. Jiang, How can I have the commit access to the svn server? I also tried the Improve this Page, which asked for user/password. The documentation seems to be wiki style. It's better to have preview of the doc when composing. The patch approach is not so convenient. regards, Qihong -- Forwarded message -- From: Bruno P. Kinoshita ki...@apache.org Date: Sun, Jul 19, 2015 at 5:52 AM Subject: Re: Final Deliveries of GSoC Project (JENA-491) To: dev@jena.apache.org dev@jena.apache.org For 1), how to commit the documentation in Jena website? If you have commit access you can either edit the pages online by clicking on the link in the top right corner Improve this Page, or check out the source repository at http://svn.apache.org/repos/asf/jena/site/trunk/ and commit your changes. Or you can check out the SVN repository, create a patch and attach it to a JIRA issue. Changes to the web site are visible initially in http://jena.staging.apache.org/ Hope that helpsBruno From: Qihong Lin confidence@gmail.com To: dev@jena.apache.org Sent: Sunday, July 19, 2015 6:17 AM Subject: Re: Final Deliveries of GSoC Project (JENA-491) Hi, I've been taking some course exams for the last week. The current project status is ahead of plan. I'll continue the project next week after the exams. For 1), how to commit the documentation in Jena website? regards, Qihong On Wed, Jul 15, 2015 at 5:43 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi Qihong, How is the project going? I've reviewed the code you committed last week. The function of Fuseki support of constructing quad is generally OK. The tests make sense to me, although just 2 tests. However, there's more deliveries for you final work. At least, 1) Formal documentation of the goals, approaches, results and etc. I suggest creating a doc page in jena website [1], in the section of Advanced SPARQL use 2) Pull requests from your code base into apache/jena ASAP Please split this up so that we can check your commits one by one. More details in the other thread advised by Andy. 3) More tests for jena-arq and jena-fuseki-core following the original framework Please let me know if anything forgotten. Best regards, Ying Jiang [1] https://jena.apache.org/documentation/query/index.html
Re: Merging GEODE-9 Spark Connector project into develop
+1 to merge On Fri, Jul 24, 2015 at 9:54 AM, Dan Smith dsm...@pivotal.io wrote: +1 to merge. -Dan On Thu, Jul 23, 2015 at 11:44 PM, Jianxia Chen jc...@pivotal.io wrote: +1 for merge On Thu, Jul 23, 2015 at 4:27 PM, Anilkumar Gingade aging...@pivotal.io wrote: +1 for merge. On Thu, Jul 23, 2015 at 10:19 AM, Jason Huynh jhu...@pivotal.io wrote: Greetings, We are hoping to merge in GEODE-9 to develop. GEODE-9 is the feature work for the gemfire/geode- spark connector. This work had previously been done on a private repo prior to Geode being in incubation and is quite large. This merge will create a sub project in Geode named gemfire-spark-connector. This project uses sbt to do it's build and has not yet been connected to the Geode build system. There will be future work to better incorporate it with the gradle as well as removing the geode jar dependency. This project has a separate set of readme/tutorial docs as well as it's own tests and integration tests. These also have not been integrated with the automated testing and will need to be included at some point. The hope was to get this merged in and do the remaining work in smaller, easier to digest chunks as well as possibly getting other contributors helping with these efforts. Currently there is a review for this entire change at: https://reviews.apache.org/r/36731/ It will probably be easier to just get a checkout of the branch to see what it looks like. Please voice any concerns, suggestions or questions on this thread. Thanks!
JENA-491 Pull Request (First Half Part)
Hi, Please confirm the pull of the first half part work of the GSoC project (JENA-491): https://github.com/apache/jena/pull/88 Note that confidencesun/master firstly is updated/merged from apache/master, and then merged from confidencesun/JENA-491. Now apache/master can pull from confidencesun/master so that the JENA-491 functions can be applied. If it's OK, I'll further deal with construct quad support of fuseki. The code have done. I can merge them in the same way. regards, Qihong
Re: Final Deliveries of GSoC Project (JENA-491)
Hi, I've been taking some course exams for the last week. The current project status is ahead of plan. I'll continue the project next week after the exams. For 1), how to commit the documentation in Jena website? regards, Qihong On Wed, Jul 15, 2015 at 5:43 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi Qihong, How is the project going? I've reviewed the code you committed last week. The function of Fuseki support of constructing quad is generally OK. The tests make sense to me, although just 2 tests. However, there's more deliveries for you final work. At least, 1) Formal documentation of the goals, approaches, results and etc. I suggest creating a doc page in jena website [1], in the section of Advanced SPARQL use 2) Pull requests from your code base into apache/jena ASAP Please split this up so that we can check your commits one by one. More details in the other thread advised by Andy. 3) More tests for jena-arq and jena-fuseki-core following the original framework Please let me know if anything forgotten. Best regards, Ying Jiang [1] https://jena.apache.org/documentation/query/index.html
[jira] [Created] (GEODE-137) Spark Connector: should connect to local GemFire server if possible
Qihong Chen created GEODE-137: - Summary: Spark Connector: should connect to local GemFire server if possible Key: GEODE-137 URL: https://issues.apache.org/jira/browse/GEODE-137 Project: Geode Issue Type: Bug Reporter: Qihong Chen Assignee: Qihong Chen DefaultGemFireConnection uses ClientCacheFactory with locator info to create ClientCache instance. In this case, the ClientCache doesn't connect to the GemFire/Geode server on the same host if there's one. This cause more network traffic and less efficient. ClientCacheFactory can create ClientCache based on GemFire server(s) info as well. Therefore, we can force the ClientCache connects to local GemFire server if possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M entries per partition)
[ https://issues.apache.org/jira/browse/GEODE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Chen resolved GEODE-120. --- Resolution: Fixed RDD.saveToGemfire() can not handle big dataset (1M entries per partition) - Key: GEODE-120 URL: https://issues.apache.org/jira/browse/GEODE-120 Project: Geode Issue Type: Sub-task Components: core, extensions Affects Versions: 1.0.0-incubating Reporter: Qihong Chen Assignee: Qihong Chen Original Estimate: 48h Remaining Estimate: 48h the connector use single region.putAll() call to save each RDD partition. But putAll() doesn't handle big dataset well (such as 1M record). Need to split the dataset into smaller chunks, and invoke putAll() for each chunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (GEODE-114) There's race condition in DefaultGemFireConnection.getRegionProxy
[ https://issues.apache.org/jira/browse/GEODE-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Chen closed GEODE-114. - There's race condition in DefaultGemFireConnection.getRegionProxy - Key: GEODE-114 URL: https://issues.apache.org/jira/browse/GEODE-114 Project: Geode Issue Type: Sub-task Components: core, extensions Affects Versions: 1.0.0-incubating Reporter: Qihong Chen Assignee: Qihong Chen Fix For: 1.0.0-incubating Original Estimate: 24h Remaining Estimate: 24h when multiple threads try to call getRegionProxy with the same region at the same time, the following exception was thrown: com.gemstone.gemfire.cache.RegionExistsException: /debs at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:2880) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2835) at com.gemstone.gemfire.cache.client.internal.ClientRegionFactoryImpl.create(ClientRegionFactoryImpl.java:223) at io.pivotal.gemfire.spark.connector.internal.DefaultGemFireConnection.getRegionProxy(DefaultGemFireConnection.scala:87) at io.pivotal.gemfire.spark.connector.internal.rdd.GemFirePairRDDWriter.write(GemFireRDDWriter.scala:47) at io.pivotal.gemfire.spark.connector.GemFirePairRDDFunctions$$anonfun$saveToGemfire$2.apply(GemFirePairRDDFunctions.scala:24) at io.pivotal.gemfire.spark.connector.GemFirePairRDDFunctions$$anonfun$saveToGemfire$2.apply(GemFirePairRDDFunctions.scala:24) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:64) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] incubator-geode git commit: GEODE-114: fix race condition in DefaultGemFireConnection.getRegionProxy
Repository: incubator-geode
Updated Branches:
refs/heads/feature/GEODE-9 60d074cc7 - 89b9aaff8
GEODE-114: fix race condition in DefaultGemFireConnection.getRegionProxy
Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/dc1c1559
Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/dc1c1559
Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/dc1c1559
Branch: refs/heads/feature/GEODE-9
Commit: dc1c1559dd8a38ffade204f5440755338de2a5c8
Parents: 2793365
Author: Qihong Chen qc...@pivotal.io
Authored: Tue Jul 14 16:49:18 2015 -0700
Committer: Qihong Chen qc...@pivotal.io
Committed: Wed Jul 15 09:44:34 2015 -0700
--
.../internal/DefaultGemFireConnection.scala | 8 ---
.../DefaultGemFireConnectionManager.scala | 24 ++--
2 files changed, 22 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/dc1c1559/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnection.scala
--
diff --git
a/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnection.scala
b/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnection.scala
index e31186b..bba6c69 100644
---
a/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnection.scala
+++
b/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnection.scala
@@ -23,8 +23,6 @@ private[connector] class DefaultGemFireConnection (
extends GemFireConnection with Logging {
private val clientCache = initClientCache()
- /** a lock object only used by getRegionProxy...() */
- private val regionLock = new Object
/** Register GemFire functions to the GemFire cluster */
FunctionService.registerFunction(RetrieveRegionMetadataFunction.getInstance())
@@ -81,7 +79,7 @@ private[connector] class DefaultGemFireConnection (
def getRegionProxy[K, V](regionPath: String): Region[K, V] = {
val region1: Region[K, V] =
clientCache.getRegion(regionPath).asInstanceOf[Region[K, V]]
if (region1 != null) region1
-else regionLock.synchronized {
+else DefaultGemFireConnection.regionLock.synchronized {
val region2 = clientCache.getRegion(regionPath).asInstanceOf[Region[K,
V]]
if (region2 != null) region2
else clientCache.createClientRegionFactory[K,
V](ClientRegionShortcut.PROXY).create(regionPath)
@@ -116,6 +114,10 @@ private[connector] class DefaultGemFireConnection (
}
}
+private[connector] object DefaultGemFireConnection {
+ /** a lock object only used by getRegionProxy...() */
+ private val regionLock = new Object
+}
/** The purpose of this class is making unit test
DefaultGemFireConnectionManager easier */
class DefaultGemFireConnectionFactory {
http://git-wip-us.apache.org/repos/asf/incubator-geode/blob/dc1c1559/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnectionManager.scala
--
diff --git
a/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnectionManager.scala
b/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnectionManager.scala
index 0463340..7495c60 100644
---
a/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnectionManager.scala
+++
b/gemfire-spark-connector/gemfire-spark-connector/src/main/scala/io/pivotal/gemfire/spark/connector/internal/DefaultGemFireConnectionManager.scala
@@ -23,17 +23,27 @@ object DefaultGemFireConnectionManager {
private[connector] val connections = mutable.Map[(String, Int),
GemFireConnection]()
/**
- * use locator host:port pair to lookup connection. create new connection
and add it
- * to `connections` if it does not exists.
+ * use locator host:port pair to lookup cached connection. create new
connection
+ * and add it to the cache `connections` if it does not exist.
*/
def getConnection(connConf: GemFireConnectionConf)
(implicit factory: DefaultGemFireConnectionFactory = new
DefaultGemFireConnectionFactory): GemFireConnection = {
-val conns = connConf.locators.map(connections withDefaultValue
null).filter(_ != null)
-if (conns.nonEmpty
[2/2] incubator-geode git commit: Merge branch 'GEODE-114' into feature/GEODE-9
Merge branch 'GEODE-114' into feature/GEODE-9 Project: http://git-wip-us.apache.org/repos/asf/incubator-geode/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-geode/commit/89b9aaff Tree: http://git-wip-us.apache.org/repos/asf/incubator-geode/tree/89b9aaff Diff: http://git-wip-us.apache.org/repos/asf/incubator-geode/diff/89b9aaff Branch: refs/heads/feature/GEODE-9 Commit: 89b9aaff835cd3545a94bd5b06ce7f0a4c8a2b4b Parents: 60d074c dc1c155 Author: Qihong Chen qc...@pivotal.io Authored: Thu Jul 16 10:11:33 2015 -0700 Committer: Qihong Chen qc...@pivotal.io Committed: Thu Jul 16 10:11:33 2015 -0700 -- .../internal/DefaultGemFireConnection.scala | 8 --- .../DefaultGemFireConnectionManager.scala | 24 ++-- 2 files changed, 22 insertions(+), 10 deletions(-) --
[jira] [Updated] (GEODE-120) RDD.saveToGemfire() can not handle big dataset (1M entries per partition)
[ https://issues.apache.org/jira/browse/GEODE-120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qihong Chen updated GEODE-120: -- Summary: RDD.saveToGemfire() can not handle big dataset (1M entries per partition) (was: RDD.saveToGemfire() can not handle big dataset (1M record per partition)) RDD.saveToGemfire() can not handle big dataset (1M entries per partition) - Key: GEODE-120 URL: https://issues.apache.org/jira/browse/GEODE-120 Project: Geode Issue Type: Sub-task Components: core, extensions Affects Versions: 1.0.0-incubating Reporter: Qihong Chen Assignee: Qihong Chen Original Estimate: 48h Remaining Estimate: 48h the connector use single region.putAll() call to save each RDD partition. But putAll() doesn't handle big dataset well (such as 1M record). Need to split the dataset into smaller chunks, and invoke putAll() for each chunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 36408: Resolve the Geode Spark Connector Build Issue
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36408/#review91625 --- Ship it! Ship It! - Qihong Chen On July 10, 2015, 11:32 p.m., Jianxia Chen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36408/ --- (Updated July 10, 2015, 11:32 p.m.) Review request for geode, anilkumar gingade, Bruce Schuchardt, Jason Huynh, and Qihong Chen. Repository: geode Description --- Remove the dependency on Pivotal's internal repo. Resolve the conflict of different netty versions on Spark and Geode. To build Geode Spark Connector, first build Geode and publish the jars to local repo. Then you build the connector using sbt. Diffs - gemfire-spark-connector/doc/1_building.md ece4a9c gemfire-spark-connector/project/Dependencies.scala 899e182 gemfire-spark-connector/project/Settings.scala ec61884 Diff: https://reviews.apache.org/r/36408/diff/ Testing --- sbt test it:test Thanks, Jianxia Chen
Re: [ANN] GSoC 2015 Project Mid-term Evaluation (PASS)
Hi, How to integrate my code into the code base? Which of the following one is the desired approach: 1) confidencesun/jena:JENA-491 - apache/jena:master 2) confidencesun/jena:JENA-491 - confidencesun/jena:master - apache/jena:master 3) confidencesun/jena:JENA-491 - apache/jena:JENA-491 - apache/jena:master regards, Qihong On Tue, Jul 7, 2015 at 1:48 AM, Andy Seaborne a...@apache.org wrote: On 05/07/15 15:49, Qihong Lin wrote: Hi, Please check my latest commits [1]. Hi Qihong, As in my other message, that has crossed over at the same time you sent this, it is time to work out the process for delivering to the Jena codebase. Let's start with changes to jena-arq. You will need to work with your mentor for this. Please can we have the discussions regularly on this list. It will make integrating the code into the code base much easier - at the moment, it's quite hard to see how that can be achieved. The latest commit has 300+ changed files almost all from merging with Jena master not your changes. I'm afraid that finding the relevant parts to review from such a large number of files is somewhat difficult. It breaks the github review tools for a start! Andy I just completed the function of QueryEngineHTTP.execConstructQuads(). You can try out the example of query_construct_quad_01() in TestQuery [2]. I'll improve the tests later on, with more questions in the other thread. Thanks! Regards, Qihong [1] https://github.com/confidencesun/jena/commit/07afdbf0fe635d41c302b25a9c51f43802ea903a [2] https://github.com/confidencesun/jena/blob/JENA-491/jena-fuseki2/jena-fuseki-core/src/test/java/org/apache/jena/fuseki/TestQuery.java On Sat, Jul 4, 2015 at 10:08 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi all, FYI, Google announced the mid-term evaluation result of GSoC 2015 project yesterday. The Jena project of Extend CONSTRUCT to build quads (JENA-491) passed the evaluation. Hi Qihong, Thanks a lot for your contribution last month! We expect your excellent work in the coming weeks. The status of the project is a little ahead of the schedule. But the rest work is also challenging. You're supposed to integrate JENA-491 into Fuseki. In addition, the tests should be improved in accordance with the original test framework of jena-arq. Have fun! Cheers, Ying Jiang
Re: [GSoC 2015 - JENA-491] JavaCC with master.jj
Hi, I've studied the jena tests. It looks like that the syntax tests are generated by syn.sh. But the execution tests are not generated by scripts, which are written by hand one by one. Is that true? Since I have enough time, I'd like to directly go for the syn.sh and syn-arq.sh to generate tests for constructing quads. Thanks! regards, Qihong On Tue, Jun 16, 2015 at 9:24 PM, Andy Seaborne a...@apache.org wrote: On 16/06/15 09:06, Qihong Lin wrote: Hi, Thanks! I just marked GRAPH mandatory, and it worked without producing the warnings. I'll look into the details later. By the way, if the new parser is ready, how to test it? I mean, where to drop the unit test code and the query strings to be tested? I'm confused with org.apache.jena.sparql.junit.QueryTest (is that what I need to deal with?). Any guideline or documentation for arq test? regards, Qihong Most testing of queries is by externally defined manifest files (manifest.ttl) jena-arq/testing/ARQ For now, keep it clean and start a new directory jena-arq/testing/ARQ/ConstructQuads with both syntax and execution tests. This is just to keep everything in one place for now. See jena-arq/testing/ARQ/Syntax/Syntax-ARQ/manifest.ttl and jena-arq/testing/ARQ/Construct/manifest.ttl. A manifest can have syntax and execution tests - it so happens that they are in separate places in the current test suite which was input the the working group. A syntax test looks like: :test_1 rdf:type mfx:PositiveSyntaxTestARQ ; dawgt:approval dawgt:NotClassified ; mf:namesyntax-select-expr-01.arq ; mf:action syntax-select-expr-01.arq ;. to parse syntax-select-expr-01.arq, expecting it to be good, and an execution test is an action and a result: :someExecutionTest rdf:type mfx:TestQuery ; mf:nameConstruct Quads 1 ; mf:action [ qt:query q-construct-1.rq ; qt:data data-1.ttl ] ; mf:result results-construct-1.ttl . an action is a query and a data file. There are different styles of layout in different places. The test suite has grown incrementally over the years of SPARQL 1.0 and SPARQL 1.1. Some test come from outside the project. You can test from the command line using the arq.qparse tool. See other message. There is a command qtest for running manifests. Background FYI: You won't need this when put everything in jena-arq/testing/ARQ/ConstructQuads but to explain: the main test syntax suites are auto-generated by syn.sh Part of that is syn-arq.sh. But hand writing syntax easier for now. Andy
Re: Where to place Spark + GemFire connector.
The problem is caused by multiple major dependencies and different release cycles. Spark Geode Connector depends on two products: Spark and Geode (not counting other dependencies), and Spark moves much faster than Geode, and some features/code are not backward compatible. Our initial connector implementation depends on Spark 1.2 in before the last week of March 15. Then Spark 1.3 was released on the last week of March, and some connector feature doesn't work with Spark 1.3, then we moved on, and now support Spark 1.3 (but not 1.2 any more, we did create tag). Two weeks ago, Spark 1.4 was released, and it breaks our connector code again. Therefore, for each Geode release, we probably need multiple Connector releases, and probably need to maintain last 2 or 3 Connector releases, for example, we need to support both Spark 1.3 and 1.4 with the current Geode code. The question is how to support this with single source repository? Thanks, Qihong
Re: [ANN] GSoC 2015 Project Mid-term Evaluation (PASS)
Hi, Please check my latest commits [1]. I just completed the function of QueryEngineHTTP.execConstructQuads(). You can try out the example of query_construct_quad_01() in TestQuery [2]. I'll improve the tests later on, with more questions in the other thread. Thanks! Regards, Qihong [1] https://github.com/confidencesun/jena/commit/07afdbf0fe635d41c302b25a9c51f43802ea903a [2] https://github.com/confidencesun/jena/blob/JENA-491/jena-fuseki2/jena-fuseki-core/src/test/java/org/apache/jena/fuseki/TestQuery.java On Sat, Jul 4, 2015 at 10:08 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi all, FYI, Google announced the mid-term evaluation result of GSoC 2015 project yesterday. The Jena project of Extend CONSTRUCT to build quads (JENA-491) passed the evaluation. Hi Qihong, Thanks a lot for your contribution last month! We expect your excellent work in the coming weeks. The status of the project is a little ahead of the schedule. But the rest work is also challenging. You're supposed to integrate JENA-491 into Fuseki. In addition, the tests should be improved in accordance with the original test framework of jena-arq. Have fun! Cheers, Ying Jiang
Extend CONSTRUCT to build quads (JENA-491)
Hi, The current W3C recommendation of SPARQL 1.1 supports the CONSTRUCT query form, which returns a single RDF graph specified by a graph template. The result is an RDF graph formed by taking each query solution in the solution sequence, substituting for the variables in the graph template, and combining the triples into a single RDF graph by set union. However, it does not directly generate quads or RDF datasets. In order to eliminate this limitation, I proposed a GSoC 2015 project [1], to extend the grammar of the CONSTRUCT query form and implement the according Jena ARQ components, which will bring more conveniences for the users manipulating RDF datasets with SPARQL. Now the code has been delivered [2], with some examples here [3]. You're welcome to try them out. Any comments are appropriated. Cheers, Qihong Lin [1] https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0/edit# [2] https://github.com/confidencesun/jena/tree/JENA-491 [3] https://github.com/confidencesun/jena/blob/JENA-491/jena-arq/src-examples/arq/examples/constructquads/ExampleConstructQuads.java
Re: [GSoC 2015 - JENA-491] Refactoring Template
On Tue, Jun 16, 2015 at 7:19 PM, Andy Seaborne a...@apache.org wrote:
On 15/06/15 11:36, Qihong Lin wrote:
Hi,
Please check my last 3 commits[1], related to refactoring Template.
master.jj is modified accordingly, *without* implementing the new
CONSTRUCT query function for quad. Just to make sure: the maven build
test for this jena-arq module is successful.
Here're some ideas for to discuss:
(1) Leaving the existing methods/constructors in-place , such as new
Template(basic graph pattern), so the source code for the SPARQL 1.0
parser does not need change.
Good!
(2) Add new constructors of Template(QuadAcc qp) for future use for
the new CONSTRUCT query for quad. I use QuadAcc here (instead of
QuadPattern), because other parts of master.jj use it for parsing
quads.
That's workable but it could be a bit neater with a builder pattern like the
current Template(BGP).
The original code had TripleCollectorBGP as the builder and when it's
finished the parsing step for the CONSTRUCT template, there is a call of
getBGP() that yields the BGP. They may not be a real copy - that's an
efficiency hack, not a design issue.
Template for quads should follow the same pattern. The constructor for
Template can take ListQuad (or add getQuadPattern to QuadAcc c.f.
TripleCollectorBGP.getBGP)
(Admittedly, the existing code should also do this in other places. Code
grows organically over a long period. Consistency is unlikely!)
Why can't you use QuadPattern(QuadAcc) in ARQ-ConstructQuery?
Andy
Because our ('GRAPH' VarOrIri)? definition is optional. It requires
more flexible than QuadPattern. In QuadPattern, 'GRAPH' token can not
be omitted.
Anything inappropriate? If it's generally OK, I'd like to continue
working on master.jj.
Great.
regards,
Qihong
[1] https://github.com/confidencesun/jena/commits/JENA-491
Re: [GSoC 2015 - JENA-491] Refactoring Template
Hi,
Here some examples to try:
https://github.com/confidencesun/jena/blob/JENA-491/jena-arq/src-examples/arq/examples/constructquads/ExampleConstructQuads.java
You can see that SPARQL 1.1 remains unchanged, both for sparql_11.jj
and the generated parser code.
regards,
Qihong
On Tue, Jun 16, 2015 at 9:04 PM, Andy Seaborne a...@apache.org wrote:
On 16/06/15 12:19, Andy Seaborne wrote:
On 15/06/15 11:36, Qihong Lin wrote:
Hi,
Please check my last 3 commits[1], related to refactoring Template.
master.jj is modified accordingly, *without* implementing the new
CONSTRUCT query function for quad. Just to make sure: the maven build
test for this jena-arq module is successful.
Is there an example that I can try?
You should be able to leave the SPARQL 1.1 version of CONSTRUCT alone. It's
only the ARQ-extended form that should have your grammar. The SPARQL 1.1
parser (like the SPARQL 1.0 parser) will be unchanged although changes in
whitespace etc in the generated java will get in the way of seeing that.
master.jj:
#ifdef ARQ
void ConstructQuery() ; { Template t ;
QuadAcc acc = new QuadAcc() ; }
...
#else
void ConstructQuery() : { Template t ;
QuadAcc acc = new QuadAcc() ; }
This second branch should be the same bytes as the existing master.jj:
void ConstructQuery() : { Template t ;
TripleCollectorBGP acc = new TripleCollectorBGP()
; }
{
...
You can test from the command line using the arq.qparse tool. Either
java -cp ... arq.qparse --syntax=ARQ --file=...
or
java -cp ... arq.qparse --file=Q.arq
The .arq extension will change it to ARQ syntax.
I'm looking forward to trying it out ...
Andy
Here're some ideas for to discuss:
(1) Leaving the existing methods/constructors in-place , such as new
Template(basic graph pattern), so the source code for the SPARQL 1.0
parser does not need change.
Good!
(2) Add new constructors of Template(QuadAcc qp) for future use for
the new CONSTRUCT query for quad. I use QuadAcc here (instead of
QuadPattern), because other parts of master.jj use it for parsing
quads.
That's workable but it could be a bit neater with a builder pattern like
the current Template(BGP).
The original code had TripleCollectorBGP as the builder and when it's
finished the parsing step for the CONSTRUCT template, there is a call of
getBGP() that yields the BGP. They may not be a real copy - that's an
efficiency hack, not a design issue.
Template for quads should follow the same pattern. The constructor for
Template can take ListQuad (or add getQuadPattern to QuadAcc c.f.
TripleCollectorBGP.getBGP)
(Admittedly, the existing code should also do this in other places. Code
grows organically over a long period. Consistency is unlikely!)
Why can't you use QuadPattern(QuadAcc) in ARQ-ConstructQuery?
Andy
Anything inappropriate? If it's generally OK, I'd like to continue
working on master.jj.
Great.
regards,
Qihong
[1] https://github.com/confidencesun/jena/commits/JENA-491
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi, Please check my commits [1]. For quick answers below: On Thu, Jun 18, 2015 at 6:18 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi Qihong, You may have already received an email from Google, which reminded us of the mid-term evaluation for the GSoC project. The evaluation will start next Friday, 26 June. The deadline to submit your evaluation is 19:00 UTC on Friday, 3 July. Please make sure not to miss it. More importantly, you're supposed to deliver your work according to the proposal before that. At least, I need to check the following output for the evaluation of your work: 1) the grammar definition: normative definition of the syntax It follows the proposal doc here [2]. 2) the new master.jj, arq.jj, with sparql_11.jj unchanged. Yeah, sparql_11.jj is now unchanged. 3) the code of new QueryExecution operations: execConstrucQuads(), execConstructDataset() Done! 4) some test code for the above, OR at least some examples/demos to prove your work Here are some examples in ExampleConstructQuads.java [3] Best regards, Ying Jiang [1] https://github.com/confidencesun/jena/commits/JENA-491 [2] https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0/edit# [3] https://github.com/confidencesun/jena/blob/JENA-491/jena-arq/src-examples/arq/examples/constructquads/ExampleConstructQuads.java regards, Qihong
Re: [GSoC 2015 - JENA-491] JavaCC with master.jj
Hi,
Thanks! I just marked GRAPH mandatory, and it worked without
producing the warnings. I'll look into the details later.
By the way, if the new parser is ready, how to test it? I mean, where
to drop the unit test code and the query strings to be tested? I'm
confused with org.apache.jena.sparql.junit.QueryTest (is that what I
need to deal with?). Any guideline or documentation for arq test?
regards,
Qihong
On Mon, Jun 15, 2015 at 11:45 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
In addition to Andy's explanation, You might take look at this
tutorial for more details on javacc lookahead:
https://javacc.java.net/doc/lookahead.html
Best regards,
Ying Jiang
On Mon, Jun 15, 2015 at 10:42 PM, Andy Seaborne a...@apache.org wrote:
Qihong,
There is an ambiguity in the grammar if you make GRAPH optional.
See rule 'Quads'
Consider these two cases:
:s :p :o .
:z { :s1 :p1 :o1 } .
:s :p :o .
:z :q :o2 .
when the parser get to end of the triple in the default graph:
:s :p :o .
there are two ways forward: more triples (TriplesTemplate) and end of the
triples part, start of named graph.
It looks ahead one token and see :z and needs to decide whether the next
rule is more triples, the :z :q :o2 . case, or the end of the triples for
the default graph and the start of a named graph the :z { :s1 :p1 :o1 } .
where it exists TriplesTemplate and moves on to QuadsNotTriples
If GRAPH then the entry to QuadsNotTriples is marked by a GRAPH which is
never in triples.
The grammar is LL(1) - a lookahead of 1 - by default.
There are two solutions (I haven't checked exact deatils):
1/ Use LOOKAHEAD(2) so it sees tokens ':z' and ':q' or ':z' (triples) and
'{' which is the named graphs case. I think this is in Quads somewhere.
2/ Leave GRAPH required.
(2) is fine for now - it will not be too unexpected to users because INSERT
DATA requires a GRAPH and it is legal TriG, even if not the short form in
TriG.
You can come back and look at (1) later. I'm keen for you to get something
going as soon as possible, not get lost in details.
Background:
There is a third solution but it's not as so simple which is to introduce an
intermediate state of MaybeTriplesMaybeQuads but if you do that, more of
the grammar needs rewriting. I'm not sure how widespread the changes would
be.
Jena's TriG parser (which is not JavaCC based see
LangTriG::oneNamedGraphBlock2)
has this comment:
// Either :s :p :o or :g { ... }
and does one look ahead to get the :s or :g (the :z above), keeps that
hanging around, does another lookahead to see '{' or not, then calls
turtle(n) if triples.
In LangTriG:
turtle() is roughly TriplesSameSubject
turtle(n) is roughly PropertyListNotEmpty
Andy
On 15/06/15 11:53, Qihong Lin wrote:
Hi,
I'm trying to play with master.jj. But the grammar script somethings
prints warning messages. The behavior is strange. In order to simplify
my question, I'd like to take the following example:
In QuadsNotTriples(), line 691 in master.jj, in the master branch:
GRAPH
If I change it to optional (which is required in future
implementations, for the new grammar):
(GRAPH)?
the grammar script goes like this:
$ ./grammar
Process grammar -- sparql_11.jj
Java Compiler Compiler Version 5.0 (Parser Generator)
(type javacc with no arguments for help)
Reading from file sparql_11.jj . . .
Warning: Choice conflict in [...] construct at line 464, column 4.
Expansion nested within construct and expansion following
construct
have common prefixes, one of which is: VAR1
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 468, column 6.
Expansion nested within construct and expansion following
construct
have common prefixes, one of which is: VAR1
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 484, column 12.
Expansion nested within construct and expansion following
construct
have common prefixes, one of which is: VAR1
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 759, column 3.
Expansion nested within construct and expansion following
construct
have common prefixes, one of which is: VAR1
Consider using a lookahead of 2 or more for nested expansion.
Warning: Choice conflict in [...] construct at line 767, column 5.
Expansion nested within construct and expansion following
construct
have common prefixes, one of which is: VAR1
Consider using a lookahead of 2 or more for nested expansion.
File TokenMgrError.java does not exist. Will create one.
File ParseException.java does not exist. Will create one.
File
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi, I just opened 2 new threads. Your comments are welcome! regards, Qihong On Mon, Jun 15, 2015 at 4:45 PM, Rob Vesse rve...@dotnetrdf.org wrote: Folks Could you please either start new threads of change the subject line appropriately for future discussions on this The generic subject line makes it hard when catching up with emails to see at a glance what this thread is about Thanks, Rob On 14/06/2015 12:28, Andy Seaborne a...@apache.org wrote: Hi, Would it be possible to add new methods constructors to Template.java while leaving the existing methods/constructors in-place (reimplemented in Template; that does not affect the SPARQL 1.0 parser). Then the source code for the SPARQL 1.0 parser does not need change. The main (and only, I think) use from the SPARQL 1.0 grammar is the constructor new Template(basic graph pattern). So if there is a constructor to match that, the generated source code for SPARQL 1.0 does not need to change (it may need to recompiled but that's happening anyway). Another constructor can take a QuadPattern. getTriple/getBGP only returns the default graph related triples. A new getQuads returns everything. Is there some code on github for the changes to Template to look at? If you drop some links here, I'll try to find some time to take a look soon. Don't worry too much at this stage - If SPARQL 1.0 gets in the way too much of you making progress, then delete it from your code (but note the fact so we can be careful when merging pull requests later!!) Andy On 14/06/15 05:16, Ying Jiang wrote: Hi Qihong, It doesn't matter. For a temperate solution, just change the code of SPARQL_10 manually to make the compiling work. Please don't wait the answers from the list. Many of the questions do not stop you from coding. We would be glad if you can deliver your code as soon/often as possible. Best regards, Ying Jiang On Fri, Jun 12, 2015 at 9:54 AM, Qihong Lin confidence@gmail.com wrote: Hi, If the checksum and grammar script is OK, I'd like to continue coding. However, the grammar script only generates ARQ and SPARQL_11, without touching SPARQL_10. If I change the code of, e.g. Template [1], it will affect SPARQL_10. Shall I modify the code of SPARQL_10 manually, change the sparql_10.jj grammar manually, or re-generate SPARQL_10 with the grammar script (and how?) ? regards, Qihong [1] https://github.com/confidencesun/jena/blob/master/jena-arq/src/main/java /org/apache/jena/sparql/syntax/Template.java On Fri, Jun 12, 2015 at 12:10 AM, Ying Jiang jpz6311...@gmail.com wrote: Hi Qihong, Have you modified any files? If the grammar file is identical, the checksum of each generated java class should be the same. Best regards, Ying Jiang On Mon, Jun 8, 2015 at 5:27 PM, Qihong Lin confidence@gmail.com wrote: Hi, The grammar has been modified for the problems you pointed out. I've tried to run grammar script to generate arq.jj, sparql_11.jj and their parser java classes, in cygwin with JavaCC 5.0. However the generated java classes are different from those in the code base: 1) ARQParser - the new generated one: public class ARQParser extends ARQParserBase implements ARQParserConstants - the old one in the code base: public class ARQParser extends ARQParserBase There's no such difference for SPARQLParser11 (both new and old ones have implements ...) 2) checksum for Token, ParseException, JavaCharStream and so on - the new generated one (Token.java): /* JavaCC - OriginalChecksum=335d1922781852977208d5cdca0fc164 (do not edit this line) */ - the old one in the code base (Token.java): /* JavaCC - OriginalChecksum=d9b4c8c9332fa3054a004615fdb22b89 (do not edit this line) */ The log from grammar script seems OK: $ ./grammar Process grammar -- sparql_11.jj Java Compiler Compiler Version 5.0 (Parser Generator) (type javacc with no arguments for help) Reading from file sparql_11.jj . . . File TokenMgrError.java does not exist. Will create one. File ParseException.java does not exist. Will create one. File Token.java does not exist. Will create one. File JavaCharStream.java does not exist. Will create one. Parser generated successfully. Create text form Java Compiler Compiler Version 5.0 (Documentation Generator Version 0.1.4) (type jjdoc with no arguments for help) Reading from file sparql_11.jj . . . Grammar documentation generated successfully in sparql_11.txt Fixing Java warnings in TokenManager ... Fixing Java warnings in Token ... Fixing Java warnings in TokenMgrError ... Fixing Java warnings in SPARQLParser11 ... Done Process grammar -- arq.jj Java Compiler Compiler Version 5.0 (Parser Generator) (type javacc with no arguments for help) Reading from file arq.jj . . . File TokenMgrError.java does not exist. Will create one. File ParseException.java does not exist. Will create one. File Token.java does not exist. Will create
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi,
If the checksum and grammar script is OK, I'd like to continue coding.
However, the grammar script only generates ARQ and SPARQL_11, without
touching SPARQL_10. If I change the code of, e.g. Template [1], it
will affect SPARQL_10. Shall I modify the code of SPARQL_10 manually,
change the sparql_10.jj grammar manually, or re-generate SPARQL_10
with the grammar script (and how?) ?
regards,
Qihong
[1]
https://github.com/confidencesun/jena/blob/master/jena-arq/src/main/java/org/apache/jena/sparql/syntax/Template.java
On Fri, Jun 12, 2015 at 12:10 AM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
Have you modified any files? If the grammar file is identical, the
checksum of each generated java class should be the same.
Best regards,
Ying Jiang
On Mon, Jun 8, 2015 at 5:27 PM, Qihong Lin confidence@gmail.com wrote:
Hi,
The grammar has been modified for the problems you pointed out.
I've tried to run grammar script to generate arq.jj, sparql_11.jj and
their parser java classes, in cygwin with JavaCC 5.0. However the
generated java classes are different from those in the code base:
1) ARQParser
- the new generated one:
public class ARQParser extends ARQParserBase implements ARQParserConstants
- the old one in the code base:
public class ARQParser extends ARQParserBase
There's no such difference for SPARQLParser11 (both new and old ones
have implements ...)
2) checksum for Token, ParseException, JavaCharStream and so on
- the new generated one (Token.java):
/* JavaCC - OriginalChecksum=335d1922781852977208d5cdca0fc164 (do not
edit this line) */
- the old one in the code base (Token.java):
/* JavaCC - OriginalChecksum=d9b4c8c9332fa3054a004615fdb22b89 (do not
edit this line) */
The log from grammar script seems OK:
$ ./grammar
Process grammar -- sparql_11.jj
Java Compiler Compiler Version 5.0 (Parser Generator)
(type javacc with no arguments for help)
Reading from file sparql_11.jj . . .
File TokenMgrError.java does not exist. Will create one.
File ParseException.java does not exist. Will create one.
File Token.java does not exist. Will create one.
File JavaCharStream.java does not exist. Will create one.
Parser generated successfully.
Create text form
Java Compiler Compiler Version 5.0 (Documentation Generator Version 0.1.4)
(type jjdoc with no arguments for help)
Reading from file sparql_11.jj . . .
Grammar documentation generated successfully in sparql_11.txt
Fixing Java warnings in TokenManager ...
Fixing Java warnings in Token ...
Fixing Java warnings in TokenMgrError ...
Fixing Java warnings in SPARQLParser11 ...
Done
Process grammar -- arq.jj
Java Compiler Compiler Version 5.0 (Parser Generator)
(type javacc with no arguments for help)
Reading from file arq.jj . . .
File TokenMgrError.java does not exist. Will create one.
File ParseException.java does not exist. Will create one.
File Token.java does not exist. Will create one.
File JavaCharStream.java does not exist. Will create one.
Parser generated successfully.
Create text form
Java Compiler Compiler Version 5.0 (Documentation Generator Version 0.1.4)
(type jjdoc with no arguments for help)
Reading from file arq.jj . . .
Grammar documentation generated successfully in arq.txt
Fixing Java warnings in TokenManager ...
Fixing Java warnings in Token ...
Fixing Java warnings in TokenMgrError ...
Fixing Java warnings in ARQParser ...
Done
Is that the expected behavior for the grammar script? Anything wrong?
regard,
Qihong
On Sat, Jun 6, 2015 at 11:05 AM, Ying Jiang jpz6311...@gmail.com wrote:
Hi,
The grammar needs revisions in some way. For example, in your
proposal, the GRAPH token can be optional. Another problem for default
graph: both { ?s :p ?o } and ?s :p ?o are valid, so QuadsNotTriples
should be re-defined.
On the other hand, you can start playing with the code of master.jj.
There's no need to wait until the grammar is ready. Your code is
supposed to be delivered as soon as possible. We can have early
feedback from the end users. Merging early will also reduce any
problems with several people changing the same file.
Best regards,
Ying Jiang
On Fri, Jun 5, 2015 at 6:25 PM, Qihong Lin confidence@gmail.com wrote:
Hi,
I added the grammar draft at the end of [1]. There're actually minor
changes on the grammar of ConstructQuery, which are marked red. Much
of the grammar from SPARQL INSERT can be reused, related to Quads. Any
comments?
regards,
Qihong
[1]
https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0
On Tue, Jun 2, 2015 at 10:10 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
Your grammar in the proposal is not formal. Why not compose a BNF/EBNF
notation one, so that others can provide more accurate comments? e.g,
the WHERE clause for the complete form and short form are quite
different. No complex
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi,
The grammar has been modified for the problems you pointed out.
I've tried to run grammar script to generate arq.jj, sparql_11.jj and
their parser java classes, in cygwin with JavaCC 5.0. However the
generated java classes are different from those in the code base:
1) ARQParser
- the new generated one:
public class ARQParser extends ARQParserBase implements ARQParserConstants
- the old one in the code base:
public class ARQParser extends ARQParserBase
There's no such difference for SPARQLParser11 (both new and old ones
have implements ...)
2) checksum for Token, ParseException, JavaCharStream and so on
- the new generated one (Token.java):
/* JavaCC - OriginalChecksum=335d1922781852977208d5cdca0fc164 (do not
edit this line) */
- the old one in the code base (Token.java):
/* JavaCC - OriginalChecksum=d9b4c8c9332fa3054a004615fdb22b89 (do not
edit this line) */
The log from grammar script seems OK:
$ ./grammar
Process grammar -- sparql_11.jj
Java Compiler Compiler Version 5.0 (Parser Generator)
(type javacc with no arguments for help)
Reading from file sparql_11.jj . . .
File TokenMgrError.java does not exist. Will create one.
File ParseException.java does not exist. Will create one.
File Token.java does not exist. Will create one.
File JavaCharStream.java does not exist. Will create one.
Parser generated successfully.
Create text form
Java Compiler Compiler Version 5.0 (Documentation Generator Version 0.1.4)
(type jjdoc with no arguments for help)
Reading from file sparql_11.jj . . .
Grammar documentation generated successfully in sparql_11.txt
Fixing Java warnings in TokenManager ...
Fixing Java warnings in Token ...
Fixing Java warnings in TokenMgrError ...
Fixing Java warnings in SPARQLParser11 ...
Done
Process grammar -- arq.jj
Java Compiler Compiler Version 5.0 (Parser Generator)
(type javacc with no arguments for help)
Reading from file arq.jj . . .
File TokenMgrError.java does not exist. Will create one.
File ParseException.java does not exist. Will create one.
File Token.java does not exist. Will create one.
File JavaCharStream.java does not exist. Will create one.
Parser generated successfully.
Create text form
Java Compiler Compiler Version 5.0 (Documentation Generator Version 0.1.4)
(type jjdoc with no arguments for help)
Reading from file arq.jj . . .
Grammar documentation generated successfully in arq.txt
Fixing Java warnings in TokenManager ...
Fixing Java warnings in Token ...
Fixing Java warnings in TokenMgrError ...
Fixing Java warnings in ARQParser ...
Done
Is that the expected behavior for the grammar script? Anything wrong?
regard,
Qihong
On Sat, Jun 6, 2015 at 11:05 AM, Ying Jiang jpz6311...@gmail.com wrote:
Hi,
The grammar needs revisions in some way. For example, in your
proposal, the GRAPH token can be optional. Another problem for default
graph: both { ?s :p ?o } and ?s :p ?o are valid, so QuadsNotTriples
should be re-defined.
On the other hand, you can start playing with the code of master.jj.
There's no need to wait until the grammar is ready. Your code is
supposed to be delivered as soon as possible. We can have early
feedback from the end users. Merging early will also reduce any
problems with several people changing the same file.
Best regards,
Ying Jiang
On Fri, Jun 5, 2015 at 6:25 PM, Qihong Lin confidence@gmail.com wrote:
Hi,
I added the grammar draft at the end of [1]. There're actually minor
changes on the grammar of ConstructQuery, which are marked red. Much
of the grammar from SPARQL INSERT can be reused, related to Quads. Any
comments?
regards,
Qihong
[1]
https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0
On Tue, Jun 2, 2015 at 10:10 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
Your grammar in the proposal is not formal. Why not compose a BNF/EBNF
notation one, so that others can provide more accurate comments? e.g,
the WHERE clause for the complete form and short form are quite
different. No complex graph patterns are allowed in the short form).
Best regards,
Ying Jiang
On Thu, May 28, 2015 at 10:59 PM, Qihong Lin confidence@gmail.com
wrote:
Hi,
Ying,
I'll stick to the list for discussion. Thanks for your guide! I
re-created a fresh new branch of JENA-491, which did not contain hp
package any more.
Andy,
You mention that the GRAPH grammar needs revisions. Please check the
following ones. I add the short form. Am I missing anything else?
Complete form:
CONSTRUCT {
# Named graph
GRAPH :g { ?s :p ?o }
# Default graph
{ ?s :p ?o }
# Named graph
:g { ?s :p ?o }
# Default graph
?s :p ?o
} WHERE { ... }
Short form:
CONSTRUCT {
} WHERE { ... }
regards,
Qihong
On Tue, May 26, 2015 at 11:12 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
As Andy mentioned, the bonding period is for community bonding, not
just mentor
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi,
I added the grammar draft at the end of [1]. There're actually minor
changes on the grammar of ConstructQuery, which are marked red. Much
of the grammar from SPARQL INSERT can be reused, related to Quads. Any
comments?
regards,
Qihong
[1]
https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0
On Tue, Jun 2, 2015 at 10:10 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
Your grammar in the proposal is not formal. Why not compose a BNF/EBNF
notation one, so that others can provide more accurate comments? e.g,
the WHERE clause for the complete form and short form are quite
different. No complex graph patterns are allowed in the short form).
Best regards,
Ying Jiang
On Thu, May 28, 2015 at 10:59 PM, Qihong Lin confidence@gmail.com wrote:
Hi,
Ying,
I'll stick to the list for discussion. Thanks for your guide! I
re-created a fresh new branch of JENA-491, which did not contain hp
package any more.
Andy,
You mention that the GRAPH grammar needs revisions. Please check the
following ones. I add the short form. Am I missing anything else?
Complete form:
CONSTRUCT {
# Named graph
GRAPH :g { ?s :p ?o }
# Default graph
{ ?s :p ?o }
# Named graph
:g { ?s :p ?o }
# Default graph
?s :p ?o
} WHERE { ... }
Short form:
CONSTRUCT {
} WHERE { ... }
regards,
Qihong
On Tue, May 26, 2015 at 11:12 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
As Andy mentioned, the bonding period is for community bonding, not
just mentor bonding. I can help you with JavaCC for the project. But
much more work needs comments and discussions with others in the
community. For example, Andy just pointed out the design issue and the
timeline changes. It's required for you to think about it and make
revisions accordingly. If any questions, just ask on the mailing list.
You'll get answers from the right one, not just from me.
It seems that your JENA-491 branch still uses com.hp.hpl.jena. Have
you re-checked out the latest code?
Best regards,
Ying Jiang
On Mon, May 25, 2015 at 11:10 PM, Andy Seaborne a...@apache.org wrote:
Hi Qihong,
One of the reasons there is a community bonding period is to create the
right social context for the work rather than just being about technical
work. As other people may be working on the areas you need to modify, this
will help avoid problems.
Discussions should be on this list and there has been nothing. Discussions
are also here to help you. It's a large codebase, and some of it quite
old.
It makes it hard to see what's important and what's not.
When it comes to decisions, the Apache catchphrase is if it isn't on the
list, it does not exist. Given that the Jena development community only
exists via this list, you'll appreciate that this is quite important.
Please discuss with Ying about regular (minimum, weekly) on the list.
There are some changes to the plan if the javacc work is done first. Alos,
we have discussed a revised design with GRAPH inside the CONSTRUCT
template.
Don't forget the case of CONSTRUCT WHERE { } shorthand form.
What does the timeline look like with these revisions? That changes the
Week 1 - Week 3 slot. Some of that freed up time should go to announcing
the changes on the users@ list, I suggest just after the mid term
evaluation.
You will have noticed that the package structure in ARQ has changed. There
is no com.hp.hpl.jena anymore, it's all moved under org.apache.jena. You
are probably better off starting from a clean clone of Jena.
Andy
On 24/05/15 14:12, Qihong Lin wrote:
Hi,
Sorry for my late response. During the community binding period, I
studied the JavaCC grammar with my mentor's help. Now, I know how to
use JavaCC in this project.
A new branch of JENA-491 [1] has just been set up. I'll code there in
the first place, and finally merge the code into jena master.
In all, I think everything is ready. Let's begin coding!
regards,
Qihong
[1] https://github.com/confidencesun/jena/tree/JENA-491
On Tue, May 19, 2015 at 4:55 AM, Andy Seaborne a...@apache.org wrote:
On 05/05/15 11:52, Andy Seaborne wrote:
On 28/04/15 13:09, Ying Jiang wrote:
Hi all,
FYI, Google announced the accepted projects of GSoC 2015 yesterday. A
Jena project of Extend CONSTRUCT to build quads (JENA-491) [1] is
among them.
Cheers,
Ying Jiang
[1]
https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0/edit#heading=h.fwbztdn0y3zl
Hi,
What's the plan for the bonding period?
Andy
Ping?
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi,
Ying,
I'll stick to the list for discussion. Thanks for your guide! I
re-created a fresh new branch of JENA-491, which did not contain hp
package any more.
Andy,
You mention that the GRAPH grammar needs revisions. Please check the
following ones. I add the short form. Am I missing anything else?
Complete form:
CONSTRUCT {
# Named graph
GRAPH :g { ?s :p ?o }
# Default graph
{ ?s :p ?o }
# Named graph
:g { ?s :p ?o }
# Default graph
?s :p ?o
} WHERE { ... }
Short form:
CONSTRUCT {
} WHERE { ... }
regards,
Qihong
On Tue, May 26, 2015 at 11:12 PM, Ying Jiang jpz6311...@gmail.com wrote:
Hi Qihong,
As Andy mentioned, the bonding period is for community bonding, not
just mentor bonding. I can help you with JavaCC for the project. But
much more work needs comments and discussions with others in the
community. For example, Andy just pointed out the design issue and the
timeline changes. It's required for you to think about it and make
revisions accordingly. If any questions, just ask on the mailing list.
You'll get answers from the right one, not just from me.
It seems that your JENA-491 branch still uses com.hp.hpl.jena. Have
you re-checked out the latest code?
Best regards,
Ying Jiang
On Mon, May 25, 2015 at 11:10 PM, Andy Seaborne a...@apache.org wrote:
Hi Qihong,
One of the reasons there is a community bonding period is to create the
right social context for the work rather than just being about technical
work. As other people may be working on the areas you need to modify, this
will help avoid problems.
Discussions should be on this list and there has been nothing. Discussions
are also here to help you. It's a large codebase, and some of it quite old.
It makes it hard to see what's important and what's not.
When it comes to decisions, the Apache catchphrase is if it isn't on the
list, it does not exist. Given that the Jena development community only
exists via this list, you'll appreciate that this is quite important.
Please discuss with Ying about regular (minimum, weekly) on the list.
There are some changes to the plan if the javacc work is done first. Alos,
we have discussed a revised design with GRAPH inside the CONSTRUCT template.
Don't forget the case of CONSTRUCT WHERE { } shorthand form.
What does the timeline look like with these revisions? That changes the
Week 1 - Week 3 slot. Some of that freed up time should go to announcing
the changes on the users@ list, I suggest just after the mid term
evaluation.
You will have noticed that the package structure in ARQ has changed. There
is no com.hp.hpl.jena anymore, it's all moved under org.apache.jena. You
are probably better off starting from a clean clone of Jena.
Andy
On 24/05/15 14:12, Qihong Lin wrote:
Hi,
Sorry for my late response. During the community binding period, I
studied the JavaCC grammar with my mentor's help. Now, I know how to
use JavaCC in this project.
A new branch of JENA-491 [1] has just been set up. I'll code there in
the first place, and finally merge the code into jena master.
In all, I think everything is ready. Let's begin coding!
regards,
Qihong
[1] https://github.com/confidencesun/jena/tree/JENA-491
On Tue, May 19, 2015 at 4:55 AM, Andy Seaborne a...@apache.org wrote:
On 05/05/15 11:52, Andy Seaborne wrote:
On 28/04/15 13:09, Ying Jiang wrote:
Hi all,
FYI, Google announced the accepted projects of GSoC 2015 yesterday. A
Jena project of Extend CONSTRUCT to build quads (JENA-491) [1] is
among them.
Cheers,
Ying Jiang
[1]
https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0/edit#heading=h.fwbztdn0y3zl
Hi,
What's the plan for the bonding period?
Andy
Ping?
Re: [ANN] GSoC 2015 Accepts a Student Project for Jena
Hi, Sorry for my late response. During the community binding period, I studied the JavaCC grammar with my mentor's help. Now, I know how to use JavaCC in this project. A new branch of JENA-491 [1] has just been set up. I'll code there in the first place, and finally merge the code into jena master. In all, I think everything is ready. Let's begin coding! regards, Qihong [1] https://github.com/confidencesun/jena/tree/JENA-491 On Tue, May 19, 2015 at 4:55 AM, Andy Seaborne a...@apache.org wrote: On 05/05/15 11:52, Andy Seaborne wrote: On 28/04/15 13:09, Ying Jiang wrote: Hi all, FYI, Google announced the accepted projects of GSoC 2015 yesterday. A Jena project of Extend CONSTRUCT to build quads (JENA-491) [1] is among them. Cheers, Ying Jiang [1] https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0/edit#heading=h.fwbztdn0y3zl Hi, What's the plan for the bonding period? Andy Ping?
Re: Fwd: Proposal submitted [JENA-491]
Hello, Please check my reply below: On Thu, Apr 9, 2015 at 12:07 PM, Ying Jiang jpz6311...@gmail.com wrote: Hi Andy, Now, we are at proposal ranking phase 1: indicate willingness to mentor, with the deadline of April 12 00:00 UTC. I've already registered as a Possible mentor for Qihong's proposal. Not sure what the next step is. Maybe, ranking all the proposals with scores. I'm waiting for the notices from the mentors@. Hi, Qihong, I think we should take Andy's advice to start discussion on dev@. The proposal has been submitted, and you can not change it. But we can put some posts below there to improve the proposal evaluation. So, I suggest: 1) Andy has made some comments about the project details on dev@ and JIRA. You can summarize them and enrich your proposal in google-melange. If you have any questions, just ask ASAP. I've modified the project proposal: https://docs.google.com/document/d/1KiDlfxMq5ZsU7vj7ZDm10yC96OZgdltwmZAZl56sTw0/edit 2) The project schedule can be adjusted, because the first half is larger so it may not split at the mid-term evaluation. 3) The javacc grammar changes should be done first. Have you learned the document of how to use javacc? Have you tried the scripts in cygwin? Yes, cygwin works for me. However I need some time to study the javacc grammar. 4) You can also set up the project documentation in Github or Jena-site. It's better to write the document of design now, like Andy says. You should know that Apache strives for a 100% success rate of GSoC project. It makes good chance to get accepted if we can get things go on ASAP. I'd like to help you with pleasure. Good luck! Best, Ying Jiang On Wed, Apr 8, 2015 at 9:18 PM, Andy Seaborne a...@apache.org wrote: Hi Ying, What is the next process step for GSoC this year? As mentor, you probably want to see this project as starting now. As you know, I don't have time to mentor this year and so can't really guarantee technical invovement at short notice. The proposal will updating for the comments; it also needs to be made consistent. This is better now as part of the submission process, not in the bonding stage. It'll improve the proposal evaluation. e.g,. the javacc grammar changes should be done first. Not much point hacking the generated java parser (it'll be over written by the grammar compiler). e.g. Documentation should arrive with a deliverable, not later (writing the document, which isn't going to be very large, helps check the design). Ying - do you want to work with Qihong to do that? As discussion on dev@? The first half is ARQ changes, the second is Fuseki changes (Fuseki2) - the first half is larger so it may not split at the mid-term evaluation. What will be important for this project is regular email traffic to dev@. It's all about making a smooth path for change. This isn't a new module, this is making changes to an area users depend on already. Several people here will want to know what's happening, hopefully comment; we should break the deliverables up to get them in piece by piece, not wait until the end. Using github and generating pull requests throughout the project will work best. There needs to be a fallback in case github is not accessible after our experiences of last year. Qihong - do you have any questions on the process? Andy On 07/04/15 15:00, Ying Jiang wrote: Hi Andy, Thanks for answering Qihong's questions! JENA-491 is not original from my idea. So, I'm very grateful if you can help the student to clarify the project goals and the scopes. Then, as the mentor, I can push the project going on when it starts, with technical assistance regarding Jena. For the first question, is it OK to expose Quad to end user in querying? I mean, we have 2 layers of Jena API: the higher one of Model/Statement/Resource, and the underlying Graph/Triple/Quad/Node. It makes sense to me if we encourage the end users using the former as much as possible. Currently, in the API, we already have: IteratorTriple QueryExecution.execConstructTriples(). I have the same doubt with it. What's your opinion? Best, Ying Jiang On Sun, Apr 5, 2015 at 2:04 AM, Andy Seaborne a...@apache.org wrote: On 03/04/15 03:47, Qihong Lin wrote: Hello Andy, It's submitted in time. Good. Ying - what is the next process step? I saw your notes, thanks. Here're some further questions. 1) API of QueryExecution Does the API look like: - IteratorQuad QueryExecution.execConstrucQuads() - Dataset QueryExecution.execConstructDataset() Both. (One builds on the other anyway.) It should mirror how execConstruct/execConstructTriples are done unless there is a very good reason not to. 2) master.jj How does master.jj generate arq.jj? What tool? You mentioned is processed with cpp. What's cpp? cpp is the C preprocessor (yes!!) It rewrites one text file to another text file. ARQ does not cpp macros, it is just
Re: Fwd: Proposal submitted [JENA-491]
Hello Andy,
It's submitted in time. I saw your notes, thanks. Here're some further
questions.
1) API of QueryExecution
Does the API look like:
- IteratorQuad QueryExecution.execConstrucQuads()
- Dataset QueryExecution.execConstructDataset()
2) master.jj
How does master.jj generate arq.jj? What tool? You mentioned is
processed with cpp. What's cpp?
3) query string sytax
I went through TriG syntax.
- For our query string, can it construct a Dataset with multiple
graphs, or just one default/named graph?
- Shall we consider using variables for named graphs? I mean ?g, not :g:
CONSTRUCT {
# Named graph
GRAPH ?g { ?s :p ?o }
} WHERE
regards,
Qihong
On Wed, Apr 1, 2015 at 9:59 PM, Andy Seaborne a...@apache.org wrote:
Hi there,
What's the status on this? Was it submitted?
I've put some notes on JENA-491 that related to the proposal about the
proposed syntax and the way to modify the grammar.
One of the important things to make sure happens is that the work is visible
on this list (confidence.sun@ is subscribed so that's a good start) through
out the project. It hard to have code appear at the end and integrate it in
time and I'm working with the assumption that there will be change to the
main codebase as a result of JENA-491 with no need for extension classes.
(I looked at the github prototype
https://github.com/confidencesun/jena/)
For example, Template class becomes quad-aware and the SPARQL 1.1 usage is
triples only because the strict syntax can't set things outside the default
graph of the datasets. Otherwise, the code is quads throughout.
So if this going ahead, we should be building community from now on, through
the bonding period and throughout the project lifetime. That includes
regular status and discussions here. Qihong - After last year with
Marmotta, I'm sure you know how things work at Apache.
Andy
On 25/03/15 13:10, Andy Seaborne wrote:
Hi Ying,
Have you signed up as a mentor yet? This is getting to be urgent.
(instructions forwarded)
On 03/25/15 01:32, Ying Jiang wrote:
Hi,
I can see it's a strong proposal. If possible, I'd be glad to be the
mentor for this project.
Andy, if other project is appropriate for Jena, I'm willing to be a
mentor, or at least a co-mentor, depending on its scopes. Hope it
helps!
The two I know of are this one from Qihong and one mention by Dimuthu [*]
The mentors signup to specific projects - it comes down to what you want
to do.
Andy
[*]
http://mail-archives.apache.org/mod_mbox/jena-dev/201503.mbox/%3CCAHzXhOimMQjHeCzdV%3Dp-jegVC-gxdPiXe%3D9QE_aGO%2BT3hfDknw%40mail.gmail.com%3E
https://issues.apache.org/jira/browse/JENA-624
Best,
Ying Jiang
On Tue, Mar 24, 2015 at 11:50 PM, Andy Seaborne a...@apache.org wrote:
On 03/24/15 13:54, Qihong Lin wrote:
Hi Andy,
I just changed the settings. Is the proposal here [1] now visible
for you?
Dr. Ying Jiang mentioned to be the potential mentor. Is that OK?
Sorry, no. I'm afraid I don't have time to be a mentor this year.
Andy
regards,
Qihong Lin
[1]
http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/confidence_sun/5639274879778816
On Tue, Mar 24, 2015 at 6:23 PM, Andy Seaborne a...@apache.org wrote:
Qihong,
We can't read that URL.
You need a mentor for the project and at the moment, Jena has very
people
with time to be a mentor this year which is severely limiting the
capacity
to take on GSoC projects.
Andy
On 24/03/15 01:51, Ying Jiang wrote:
Hi,
I can not open your proposal. Maybe you can mark it public to
everyone?
Although I can give some advice, you'd better also ask for feedback
from other committers in the Jena community. It's vital to make clear
the project details through discussion in the dev mailing list before
GSoC project begins. Guys there are nice and helpful. So, just
subscribe to it and post your ideas for comments. Good luck!
Ying Jiang
-- Forwarded message --
From: Qihong Lin confidence@gmail.com
Date: Tue, Mar 24, 2015 at 12:39 AM
Subject: Proposal submitted
To: Ying Jiang jpz6311...@gmail.com
Hello, Dr. Jiang
I've submitted a project proposal for GSoC 2015. Could you please
help
me improve it? Thanks.
http://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015/confidence_sun/5639274879778816
regards,
Qihong
Re: Fwd: Proposal submitted
Hi Andy, I just changed the settings. Is the proposal here [1] now visible for you? Dr. Ying Jiang mentioned to be the potential mentor. Is that OK? regards, Qihong Lin [1] http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/confidence_sun/5639274879778816 On Tue, Mar 24, 2015 at 6:23 PM, Andy Seaborne a...@apache.org wrote: Qihong, We can't read that URL. You need a mentor for the project and at the moment, Jena has very people with time to be a mentor this year which is severely limiting the capacity to take on GSoC projects. Andy On 24/03/15 01:51, Ying Jiang wrote: Hi, I can not open your proposal. Maybe you can mark it public to everyone? Although I can give some advice, you'd better also ask for feedback from other committers in the Jena community. It's vital to make clear the project details through discussion in the dev mailing list before GSoC project begins. Guys there are nice and helpful. So, just subscribe to it and post your ideas for comments. Good luck! Ying Jiang -- Forwarded message -- From: Qihong Lin confidence@gmail.com Date: Tue, Mar 24, 2015 at 12:39 AM Subject: Proposal submitted To: Ying Jiang jpz6311...@gmail.com Hello, Dr. Jiang I've submitted a project proposal for GSoC 2015. Could you please help me improve it? Thanks. http://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015/confidence_sun/5639274879778816 regards, Qihong
Is possible to invoke updateStateByKey twice on the same RDD
I need to implement following logic in a spark streaming app: for the incoming dStream, do some transformation, and invoke updateStateByKey to update state object for each key (mark data entries that are updated as dirty for next step), then let state objects produce event(s) based (based on dirty flag). After that, need to update the state object to clean up the dirty flags so it won't produce the same event(s) in next batch. The pseudo code is followings: 1. val dstream0 = ... 2. val state = dstream0.updateStateByKey[...](...) 3. val events = state.flatMapValues(stateobj = stateobj.produceEvents(...)) 4. ?? how to update state again ?? Is it possible to do this? If so, how? Or is there alternative way to archive the same thing? I tried to update state object in stateobj.produceEvents(...) method, but the state object at line#2 in next batch doesn't contain the change made in line#3 (in previous batch). Any suggestion? Thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-possible-to-invoke-updateStateByKey-twice-on-the-same-RDD-tp16107.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RE: Change RDDs using map()
if you want the result as RDD of (key, 1) new_rdd = rdd.filter(x = x._2 == 1) if you want result as RDD of keys (since you know the values are 1), then new_rdd = rdd.filter(x = x._2 == 1).map(x = x._1) x._1 and x._2 are the way of scala to access the key and value from key/value pair. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Change-RDDs-using-map-tp14436p14481.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Dealing with Time Series Data
what are you trying to do? generate time series from your data in HDFS, or doing some transformation and/or aggregation from your time series data in HDFS? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Dealing-with-Time-Series-Data-tp14275p14482.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark Streaming - batchDuration for streaming
Here's official spark document about batch size/interval: http://spark.apache.org/docs/latest/streaming-programming-guide.html#setting-the-right-batch-size spark is batch oriented processing. As you mentioned, the streaming is continuous flow, and core spark can not handle it. Spark streaming bridges the gap between the continuous flow and batch oriented processing. It generates an RDD from continuous data flow/stream every batch interval, then the spark can process them as normal RDDs. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-batchDuration-for-streaming-tp14469p14487.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to initialize StateDStream
there's no need to initialize StateDStream. Take a look at example StatefulNetworkWordCount.scala, it's part of spark source code. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-initialize-StateDStream-tp14113p14146.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to initialize StateDStream
I'm not sure what you mean by previous run. Is it previous batch? or previous run of spark-submit? If it's previous batch (spark streaming creates a batch every batch interval), then there's nothing to do. If it's previous run of spark-submit (assuming you are able to save the result somewhere), then I can think of two possible ways to do it: 1. read saved result as RDD (just do this once), and join the RDD with each RDD of the stateStream. 2. add extra logic to updateFunction: when the previous state is None (one of two Option type values), you get save state for the given key from saved result somehow, then your original logic to create new state object based on Seq[V] and previous state. note that you need use this version of updateFunction: updateFunc: (Iterator[(K, Seq[V], Option[S])]) = Iterator[(K, S)], which make key available to the update function. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-initialize-StateDStream-tp14113p14176.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: compiling spark source code
follow the instruction here: http://spark.apache.org/docs/latest/building-with-maven.html -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/compiling-spark-source-code-tp13980p14144.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RE: how to setup steady state stream partitions
Thanks for your response! I found that too, and it does the trick! Here's refined code: val inputDStream = ... val keyedDStream = inputDStream.map(...) // use sensorId as key val partitionedDStream = keyedDstream.transform(rdd = rdd.partitionBy(new MyPartitioner(...))) val stateDStream = partitionedDStream.updateStateByKey[...](udpateFunction, new MyPartitioner(...)) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-setup-steady-state-stream-partitions-tp13850p13931.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to set java.library.path in a spark cluster
Add something like following to spark-env.sh export LD_LIBRARY_PATH=path of libmosekjava7_0.so:$LD_LIBRARY_PATH (and remove all 5 exports you listed). Then restart all worker nodes, and try again. Good luck! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-set-java-library-path-in-a-spark-cluster-tp13854p13857.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: how to choose right DStream batch interval
repost since original msg was marked with This post has NOT been accepted by the mailing list yet. I have some questions regarding DStream batch interval: 1. if it only take 0.5 second to process the batch 99% of time, but 1% of batches need 5 seconds to process (due to some random factor or failures), then what's the right batch interval? 5 seconds (the worst case)? 2. What will happen to DStream processing if 1 batch took longer than batch interval? Can Spark recover from that? Thanks, Qihong -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-choose-right-DStream-batch-interval-tp13578p13579.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Running spark-shell (or queries) over the network (not from master)
the command should be spark-shell --master spark://master ip on EC2:7077. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-spark-shell-or-queries-over-the-network-not-from-master-tp13543p13593.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Running spark-shell (or queries) over the network (not from master)
Since you are using your home computer, so it's probably not reachable by EC2 from internet. You can try to set spark.driver.host to your WAN ip, spark.driver.port to a fixed port in SparkConf, and open that port in your home network (port forwarding to the computer you are using). see if that helps. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-spark-shell-or-queries-over-the-network-not-from-master-tp13543p13595.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: [GSoC] less 1 month left
Dear Sergio, I'm sorry I can't understand you questions. It's appreciated if you can help me clarify these: 1) If what changes in the new alternative CDI service is just the invocations to the RepositoryConnection (I'd need to check the diff to really assert that), would it still make sense to you to provide an alternative CDIservice? What do you mean by the new alternative CDI service, and an alternative CDIservice? I'm now translating the CDI service (LdpServiceImpl) into SPARQL implementation. If anything changes, I can translate it accordingly. Is that enough? 2) Or would be more maintainable to move the translation into an RepositoryConnection wrapper or something like that? What's an RepositoryConnection wrapper? Why do we need that (i.e. why more maintainable )? Can you show me an example? Thanks! Best regards, Qihong Lin On Mon, Jul 28, 2014 at 9:00 PM, Sergio Fernández wik...@apache.org wrote: Dear Qihong Lin, On 27/07/14 12:41, Qihong Lin wrote: I'm making documentation in git wiki: [1] https://github.com/confidencesun/marmotta/wiki/GSoC-2014-Project-(MARMOTTA-444) [2] https://github.com/confidencesun/marmotta/wiki/Developer-Documentation [3] https://github.com/confidencesun/marmotta/wiki/User-Documentation Perfect. Try to evolve as much as possible that documentation to start to prepare the final report. A conclusions chapter from your since would be nice. Think about what aspects are interesting to discuss. Taking a quick look to dev documentation, there is one aspect I'd like to hear from you about the mapping of method. If what changes in the new alternative CDI service is just the invocations to the RepositoryConnection (I'd need to check the diff to really assert that), would it still make sense to you to provide an alternative CDIservice? Or would be more maintainable to move the translation into an RepositoryConnection wrapper or something like that? Think about that, write down your conclusions, and ideally provide a proof-of-concept implementation. Currently, the SPARQL implementation is up to date to the ldp and develop branches. I'll make the translations soon after any new changes from upstreams, and report to you in time. Probably we'll finally do it early next week. I'll keep you informed. Cheers, -- Sergio Fernández Partner Technology Manager Redlink GmbH m: +43 660 2747 925 e: sergio.fernan...@redlink.co w: http://redlink.co
Re: [GSoC] less 1 month left
Dear Sergio, I'm making documentation in git wiki: [1] https://github.com/confidencesun/marmotta/wiki/GSoC-2014-Project-(MARMOTTA-444) [2] https://github.com/confidencesun/marmotta/wiki/Developer-Documentation [3] https://github.com/confidencesun/marmotta/wiki/User-Documentation Currently, the SPARQL implementation is up to date to the ldp and develop branches. I'll make the translations soon after any new changes from upstreams, and report to you in time. Best regards, Qihong Lin On Thu, Jul 24, 2014 at 9:59 PM, Sergio Fernández wik...@apache.org wrote: Dear Qihong Lin, On 24/07/14 14:06, Qihong Lin wrote: Thanks for your reminder! I've been watching the marmotta's branch refs/heads/develop (not much changes related to ldp service these days). As you pointed out, I just checked the ldp branch and updated my LdpServiceSPARQLImpl accordingly [1]. Now both LdpServiceSPARQLImpl and LdpServiceImpl passed my unit tests as expected. It'd be great if you can report the current status of the SPARQL implementation, as well as periodically report your progress. I'll keep a close eye on both the develop and ldp branches in the coming weeks. With previous experience in the first half part of the project, I think it's not difficult to translate the remaining changes to the SPARQL implementations. I'll check and translate the changes everyday. After the GSoC project, I can also offer contribution related to this work. That'd be great! That's one of the goals of GSoC is not always explored by the student. Hope you can continue collaborating with us after the program ends. In the meanwhile, I'll work on the documentations and the tests (performance, w3c test suite, etc), as you mentioned in the mid-term evaluation. I've been studying JMeter, TestNG and Rest-Assured. I'll have a try in the next week and keep you posted. Where are you documenting all your stuff? As I explained you, I'd appreciate to have access to it in advance. Cheers, -- Sergio Fernández Partner Technology Manager Redlink GmbH m: +43 660 2747 925 e: sergio.fernan...@redlink.co w: http://redlink.co
Re: [GSoC] less 1 month left
Dear Sergio, Thanks for your reminder! I've been watching the marmotta's branch refs/heads/develop (not much changes related to ldp service these days). As you pointed out, I just checked the ldp branch and updated my LdpServiceSPARQLImpl accordingly [1]. Now both LdpServiceSPARQLImpl and LdpServiceImpl passed my unit tests as expected. I'll keep a close eye on both the develop and ldp branches in the coming weeks. With previous experience in the first half part of the project, I think it's not difficult to translate the remaining changes to the SPARQL implementations. I'll check and translate the changes everyday. After the GSoC project, I can also offer contribution related to this work. In the meanwhile, I'll work on the documentations and the tests (performance, w3c test suite, etc), as you mentioned in the mid-term evaluation. I've been studying JMeter, TestNG and Rest-Assured. I'll have a try in the next week and keep you posted. Best, Qihong Lin [1] https://github.com/confidencesun/marmotta/commit/cff11e06c0c13bd8d8a10fea5e9eccf8d31f86ba On Wed, Jul 23, 2014 at 3:45 PM, Sergio Fernández wik...@apache.org wrote: According the official time-line: http://www.google-melange.com/gsoc/events/google/gsoc2014 August 18th is the official 'pencils down' date for GSoC this year. Please, Qihong Lin, remember to continue pushing hard. I did not see some much activity or reporting in the last weeks, and you still have so much to do for achieving the goals of the project. We had been working at the ldp branch, fixing details discovered by the W3C test suite. They mainly affect the web service layer, but I think we extended the service api with some new methods. Next week Jakob and myself we plan to continue working on that, with the idea of merging stable changes back to develop during the week. Please, stay tune to the changes, and decide whether you can afford or not to adopt them at this stage. Personally I'll offline 6th to 20th, so everything that you deliver before would make my final evaluation much easier. Thanks for your contribution to the Apache Marmotta project. Cheers, -- Sergio Fernández Partner Technology Manager Redlink GmbH m: +43 660 2747 925 e: sergio.fernan...@redlink.co w: http://redlink.co
