Christopher Johnson (WMDE) has submitted this change and it was merged.
Change subject: fixed untracked files
......................................................................
fixed untracked files
Change-Id: Idadb228e44c948719929101b24e65227441eb0de
---
M .gitignore
D data/sparql/spql1.tsv
D data/sparql/spql10.tsv
D data/sparql/spql11.tsv
D data/sparql/spql12.tsv
D data/sparql/spql13.tsv
D data/sparql/spql14.tsv
D data/sparql/spql15.tsv
D data/sparql/spql16.tsv
D data/sparql/spql17.tsv
D data/sparql/spql18.tsv
D data/sparql/spql19.tsv
D data/sparql/spql2.tsv
D data/sparql/spql20.tsv
D data/sparql/spql21.tsv
D data/sparql/spql3.tsv
D data/sparql/spql4.tsv
D data/sparql/spql5.tsv
D data/sparql/spql6.tsv
D data/sparql/spql7.tsv
D data/sparql/spql8.tsv
D data/sparql/spql9.tsv
M src/scripts/bulk_sparql.Rout
23 files changed, 35 insertions(+), 100 deletions(-)
Approvals:
Christopher Johnson (WMDE): Verified; Looks good to me, approved
diff --git a/.gitignore b/.gitignore
index e70988f..3f10779 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,4 +2,4 @@
.Rproj.user/
.RData
.Rproj.user
-/data/sparql/*
+data/sparql/
diff --git a/data/sparql/spql1.tsv b/data/sparql/spql1.tsv
deleted file mode 100644
index abce504..0000000
--- a/data/sparql/spql1.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 2353341
-2015-10-18 "2326987"
-2015-10-20 "2327322"
diff --git a/data/sparql/spql10.tsv b/data/sparql/spql10.tsv
deleted file mode 100644
index ceec31f..0000000
--- a/data/sparql/spql10.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 887
-2015-10-18 "886"
-2015-10-20 "886"
diff --git a/data/sparql/spql11.tsv b/data/sparql/spql11.tsv
deleted file mode 100644
index 9848c32..0000000
--- a/data/sparql/spql11.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 31
-2015-10-18 "31"
-2015-10-20 "31"
diff --git a/data/sparql/spql12.tsv b/data/sparql/spql12.tsv
deleted file mode 100644
index 0ed782d..0000000
--- a/data/sparql/spql12.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 30
-2015-10-18 "30"
-2015-10-20 "30"
diff --git a/data/sparql/spql13.tsv b/data/sparql/spql13.tsv
deleted file mode 100644
index 3323471..0000000
--- a/data/sparql/spql13.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 2513357
-2015-10-18 "2507958"
-2015-10-20 "2507730"
diff --git a/data/sparql/spql14.tsv b/data/sparql/spql14.tsv
deleted file mode 100644
index 7e7a699..0000000
--- a/data/sparql/spql14.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 1121470
-2015-10-18 "1120407"
-2015-10-20 "1120855"
diff --git a/data/sparql/spql15.tsv b/data/sparql/spql15.tsv
deleted file mode 100644
index 9a895b0..0000000
--- a/data/sparql/spql15.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 1673007
-2015-10-18 "1673030"
-2015-10-20 "1678643"
diff --git a/data/sparql/spql16.tsv b/data/sparql/spql16.tsv
deleted file mode 100644
index 2e1d87d..0000000
--- a/data/sparql/spql16.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 2113063
-2015-10-18 "2112502"
-2015-10-20 "2113112"
diff --git a/data/sparql/spql17.tsv b/data/sparql/spql17.tsv
deleted file mode 100644
index 080eada..0000000
--- a/data/sparql/spql17.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 2921028
-2015-10-18 "2921747"
-2015-10-20 "2925817"
diff --git a/data/sparql/spql18.tsv b/data/sparql/spql18.tsv
deleted file mode 100644
index 2400d77..0000000
--- a/data/sparql/spql18.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 925204
-2015-10-18 "925324"
-2015-10-20 "925546"
diff --git a/data/sparql/spql19.tsv b/data/sparql/spql19.tsv
deleted file mode 100644
index f754450..0000000
--- a/data/sparql/spql19.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 24426
-2015-10-18 "24422"
-2015-10-20 "24425"
diff --git a/data/sparql/spql2.tsv b/data/sparql/spql2.tsv
deleted file mode 100644
index 3a94b7d..0000000
--- a/data/sparql/spql2.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 195633
-2015-10-18 "195419"
-2015-10-20 "195743"
diff --git a/data/sparql/spql20.tsv b/data/sparql/spql20.tsv
deleted file mode 100644
index bc65691..0000000
--- a/data/sparql/spql20.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 given name
-2015-10-18 "given name"
-2015-10-20 "given name"
diff --git a/data/sparql/spql21.tsv b/data/sparql/spql21.tsv
deleted file mode 100644
index 6d43b6a..0000000
--- a/data/sparql/spql21.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 2469638
-2015-10-18 "2471593"
-2015-10-20 "2488800"
diff --git a/data/sparql/spql3.tsv b/data/sparql/spql3.tsv
deleted file mode 100644
index 7d0e0a0..0000000
--- a/data/sparql/spql3.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 67683
-2015-10-18 "66813"
-2015-10-20 "67958"
diff --git a/data/sparql/spql4.tsv b/data/sparql/spql4.tsv
deleted file mode 100644
index 023a67a..0000000
--- a/data/sparql/spql4.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 47367
-2015-10-18 "44710"
-2015-10-20 "44854"
diff --git a/data/sparql/spql5.tsv b/data/sparql/spql5.tsv
deleted file mode 100644
index 1df29f8..0000000
--- a/data/sparql/spql5.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 1872
-2015-10-18 "1867"
-2015-10-20 "1867"
diff --git a/data/sparql/spql6.tsv b/data/sparql/spql6.tsv
deleted file mode 100644
index 8d8a1ac..0000000
--- a/data/sparql/spql6.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 649
-2015-10-18 "648"
-2015-10-20 "648"
diff --git a/data/sparql/spql7.tsv b/data/sparql/spql7.tsv
deleted file mode 100644
index 6612869..0000000
--- a/data/sparql/spql7.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 35
-2015-10-18 "32"
-2015-10-20 "32"
diff --git a/data/sparql/spql8.tsv b/data/sparql/spql8.tsv
deleted file mode 100644
index fcca021..0000000
--- a/data/sparql/spql8.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 20
-2015-10-18 "20"
-2015-10-20 "20"
diff --git a/data/sparql/spql9.tsv b/data/sparql/spql9.tsv
deleted file mode 100644
index 0448087..0000000
--- a/data/sparql/spql9.tsv
+++ /dev/null
@@ -1,4 +0,0 @@
-Sys.Date.. text
-2015-10-17 205
-2015-10-18 "205"
-2015-10-20 "205"
diff --git a/src/scripts/bulk_sparql.Rout b/src/scripts/bulk_sparql.Rout
index a387369..53f4929 100644
--- a/src/scripts/bulk_sparql.Rout
+++ b/src/scripts/bulk_sparql.Rout
@@ -17,10 +17,15 @@
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
-[Previously saved workspace restored]
-
> #Bulk Query of WDQS and write to TSV
-> source("config.R")
+> source("/srv/dashboards/shiny-server/wdm/src/config.R")
+
+Attaching package: ‘shinydashboard’
+
+The following object is masked from ‘package:graphics’:
+
+ box
+
Loading required package: zoo
Attaching package: ‘zoo’
@@ -55,10 +60,6 @@
Attaching package: ‘data.table’
-The following object is masked _by_ ‘.GlobalEnv’:
-
- .N
-
The following objects are masked from ‘package:lubridate’:
hour, mday, month, quarter, wday, week, yday, year
@@ -71,9 +72,16 @@
last
->
+
+Attaching package: ‘DT’
+
+The following objects are masked from ‘package:shiny’:
+
+ dataTableOutput, renderDataTable
+
> output_path = "/srv/dashboards/shiny-server/wdm/data/sparql/"
-> qlist <- read_file("./assets/rdfq.xml")
+> qlist <- read_file("/srv/dashboards/shiny-server/wdm/assets/rdfq.xml")
+>
> rdfq <- xmlParse(qlist)
> queries <- xmlToDataFrame(nodes = getNodeSet(rdfq, "//rdfq:select", c(rdfq =
> "http://wikiba.se/rdfq#";)))
> prefixes <- xmlToDataFrame(nodes = getNodeSet(rdfq, "//rdfq:prefix", c(rdfq
> = "http://wikiba.se/rdfq#";)))
@@ -87,24 +95,35 @@
+ return(result)
+ }
>
-> write_tsv <- function(x, filename){
+> write_tsv <- function(result, filename){
++ date = Sys.Date()
+ file_uri <- paste0(output_path, filename)
-+ out = data.frame(Sys.Date(), x)
++ out = data.frame(date, result)
+ write.table(out, file=file_uri, append = TRUE, sep = "\t", row.names =
FALSE, col.names = FALSE)
+ }
>
> bulk_sparql_query <- function(esc_queries) {
+ for(q in esc_queries) {
-+ x <- get_sparql_result(wdqs_uri, pfx, q)
-+ tsv_file <- paste0("spql", match(q, esc_queries), ".tsv")
-+ write_tsv(x, tsv_file)
++ result <- get_sparql_result(wdqs_uri, pfx, q)
++ tsv_filename <- paste0("spql", match(q, esc_queries), ".tsv")
++ write_tsv(result, tsv_filename)
+ }
+ }
>
> esc_queries <- lapply(queries$text, curl_escape)
> pfx <- paste(prefixes$text, collapse="")
> bulk_sparql_query (esc_queries)
+Warning messages:
+1: closing unused connection 11
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20wikibase%3ApropertyType%20wikibase%3AQuantity%7D)
+2: closing unused connection 10
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20wikibase%3ApropertyType%20wikibase%3AMonolingualtext%7D)
+3: closing unused connection 9
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20wikibase%3ApropertyType%20wikibase%3ACommonsMedia%7D)
+4: closing unused connection 8
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20wikibase%3ApropertyType%20wikibase%3AWikibaseItem%7D)
+5: closing unused connection 7
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20wikibase%3Aqualifier%20%3Fo%7D)
+6: closing unused connection 6
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20wikibase%3Arank%20wikibase%3APreferredRank%7D)
+7: closing unused connection 5
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20a%20wikibase%3AQuantityValue%7D)
+8: closing unused connection 4
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20a%20wikibase%3ATimeValue%7D)
+9: closing unused connection 3
(https://query.wikidata.org/bigdata/namespace/wdq/sparql?query=PREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3EPREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3EPREFIX%20wd%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fentity%2F%3EPREFIX%20rdfs%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3EPREFIX%20prov%3A%20%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3EPREFIX%20wdref%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Freference%2F%3ESELECT%20%28count%28distinct%28%3Fs%29%29%20AS%20%3Fscount%29%20WHERE%20%7B%3Fs%20%3Fp%20wikibase%3AGlobecoordinateValue%7D)
>
> proc.time()
user system elapsed
- 30.308 27.488 57.795
+ 29.516 25.080 54.621
--
To view, visit https://gerrit.wikimedia.org/r/247809
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Idadb228e44c948719929101b24e65227441eb0de
Gerrit-PatchSet: 1
Gerrit-Project: wikidata/analytics/dashboard
Gerrit-Branch: master
Gerrit-Owner: Christopher Johnson (WMDE) <christopher.john...@wikimedia.de>
Gerrit-Reviewer: Christopher Johnson (WMDE) <christopher.john...@wikimedia.de>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
