Hi all, We just released Scio 0.7.0. This release is based on the latest Beam 2.9.0, includes a lot of improvements, bug fixes, and some breaking changes. To have a smooth transition to this new version, make sure you have a look at the v0.7.0 Migration Guide <https://spotify.github.com/scio/migrations/v0.7.0-Migration-Guide>.
Cheers, Regadas https://github.com/spotify/scio/releases/tag/v0.7.0 <https://github.com/spotify/scio/releases/tag/v0.7.0> "Suricata suricatta" Breaking changes See v0.7.0 Migration Guide <https://spotify.github.com/scio/migrations/v0.7.0-Migration-Guide> for detailed instructions New Magnolia <https://github.com/propensive/magnolia> based Coders <https://spotify.github.com/scio/internals/ScioIO> derivation replaces ClassTag and Kryo New ScioIO <https://spotify.github.com/scio/internals/ScioIO> replaces TestIO[T] to simplify IO implementation and stubbing in JobTest Update dynamic file destination API #1305 <https://github.com/spotify/scio/issues/1305> Remove deprecated TensorFlow graph prediction method #1370 <https://github.com/spotify/scio/pull/1370> Object file IO is no longer backwards compatible due to coder changes Refactor bigquery client (#1439 <https://github.com/spotify/scio/pull/1439>) Features Macro based coder derivation for magnitudes faster (de)serialization (#1454 <https://github.com/spotify/scio/pull/1454>)(#1394 <https://github.com/spotify/scio/pull/1394>)(#1440 <https://github.com/spotify/scio/pull/1440>)(#1434 <https://github.com/spotify/scio/pull/1434>)(#1427 <https://github.com/spotify/scio/pull/1427>)(#1412 <https://github.com/spotify/scio/pull/1412>)(#1401 <https://github.com/spotify/scio/pull/1401>)(#1429 <https://github.com/spotify/scio/pull/1429>)(#1554 <https://github.com/spotify/scio/pull/1554>)(#1494 <https://github.com/spotify/scio/pull/1494>)(#1438 <https://github.com/spotify/scio/pull/1438>)(#1605 <https://github.com/spotify/scio/pull/1605>)(#1612 <https://github.com/spotify/scio/pull/1612>) Redesigned unified ScioIO[T] for all IO modules Add SCollection#{readAll,readAllBytes} (splittable DoFn support) #796 <https://github.com/spotify/scio/issues/796> #1363 <https://github.com/spotify/scio/pull/1363> Sparse lookups (#1398 <https://github.com/spotify/scio/pull/1398>)(#1354 <https://github.com/spotify/scio/issues/1354>)(#1393 <https://github.com/spotify/scio/pull/1393>) Add sparse left and right outer joins #1386 <https://github.com/spotify/scio/pull/1386> Check and warn chained joins #1362 <https://github.com/spotify/scio/pull/1362> Support Parquet compression #1189 <https://github.com/spotify/scio/issues/1189> #1318 <https://github.com/spotify/scio/pull/1318> Port Parquet IO to Parquet 1.10 #1340 <https://github.com/spotify/scio/issues/1340> #1345 <https://github.com/spotify/scio/pull/1345> Configurable fetch and batch size for JDBC IO #1314 <https://github.com/spotify/scio/pull/1314> Add PubSubIO batch size write params (#1433 <https://github.com/spotify/scio/pull/1433>) Improve coders messages Add BigQueryType typesafe args (#1476 <https://github.com/spotify/scio/pull/1476>)(#1431 <https://github.com/spotify/scio/pull/1431>) support PubsubMessage in PubsubIO (#1395 <https://github.com/spotify/scio/pull/1395>) Add subscription function to PubSubAdmin(#1483 <https://github.com/spotify/scio/pull/1483>) Register sys.props (#1404 <https://github.com/spotify/scio/pull/1404>)(#1406 <https://github.com/spotify/scio/pull/1406>) Make typed and default args parsing logic more test friendly (#1421 <https://github.com/spotify/scio/pull/1421>) Add Google Spanner package (#1491 <https://github.com/spotify/scio/pull/1491>) Add BigQuery TimePartitioning support, fix #1419 <https://github.com/spotify/scio/issues/1419> (#1466 <https://github.com/spotify/scio/pull/1466>) Add Numeric type support in scio-bigquery (#1599 <https://github.com/spotify/scio/pull/1599>) Add scalafix rules (#1435 <https://github.com/spotify/scio/pull/1435>)(#1464 <https://github.com/spotify/scio/pull/1464>)(#1474 <https://github.com/spotify/scio/pull/1474>)(#1468 <https://github.com/spotify/scio/pull/1468>)(#1470 <https://github.com/spotify/scio/pull/1470>) Expose transform function (#1492 <https://github.com/spotify/scio/pull/1492>)(#1487 <https://github.com/spotify/scio/pull/1487>) Allow creating DataflowResult from df Job (#1481 <https://github.com/spotify/scio/pull/1481>) Remove Future.failed in IOs (#1482 <https://github.com/spotify/scio/pull/1482>) Add better error messages when missing sys.props (#1488 <https://github.com/spotify/scio/pull/1488>)(#1461 <https://github.com/spotify/scio/pull/1461>) Avoid second sql legacy check when using extractTables query op (#1508 <https://github.com/spotify/scio/pull/1508>) Add support for more WriteDisposition's in bigquery writeRows (#1511 <https://github.com/spotify/scio/issues/1511>) Add call site transform name in union all (#1499 <https://github.com/spotify/scio/pull/1499>) Update apache beam to 2.9.0 (#1580 <https://github.com/spotify/scio/pull/1580>) Updated other dependencies (#1589 <https://github.com/spotify/scio/pull/1589>)(#1586 <https://github.com/spotify/scio/pull/1586>)(#1578 <https://github.com/spotify/scio/pull/1578>)(#1579 <https://github.com/spotify/scio/pull/1579>)(#1489 <https://github.com/spotify/scio/pull/1489>)(#1544 <https://github.com/spotify/scio/pull/1544>)(#1520 <https://github.com/spotify/scio/pull/1520>)(#1539 <https://github.com/spotify/scio/pull/1539>)(#1534 <https://github.com/spotify/scio/pull/1534>)(#1533 <https://github.com/spotify/scio/pull/1533>)(#1512 <https://github.com/spotify/scio/pull/1512>)(#1517 <https://github.com/spotify/scio/pull/1517>)(#1531 <https://github.com/spotify/scio/pull/1531>)(#1521 <https://github.com/spotify/scio/pull/1521>)(#1532 <https://github.com/spotify/scio/pull/1532>)(#1538 <https://github.com/spotify/scio/pull/1538>)(#1540 <https://github.com/spotify/scio/pull/1540>)(#1526 <https://github.com/spotify/scio/pull/1526>)(#1529 <https://github.com/spotify/scio/pull/1529>)(#1518 <https://github.com/spotify/scio/pull/1518>)(#1519 <https://github.com/spotify/scio/pull/1519>)(#1536 <https://github.com/spotify/scio/pull/1536>)(#1513 <https://github.com/spotify/scio/pull/1513>)(#1530 <https://github.com/spotify/scio/pull/1530>)(#1535 <https://github.com/spotify/scio/pull/1535>)(#1527 <https://github.com/spotify/scio/pull/1527>)(#1525 <https://github.com/spotify/scio/pull/1525>)(#1524 <https://github.com/spotify/scio/pull/1524>)(#1523 <https://github.com/spotify/scio/pull/1523>)(#1514 <https://github.com/spotify/scio/pull/1514>)(#1515 <https://github.com/spotify/scio/pull/1515>)(#1516 <https://github.com/spotify/scio/pull/1516>)(#1537 <https://github.com/spotify/scio/pull/1537>)(#1509 <https://github.com/spotify/scio/pull/1509>)(#1510 <https://github.com/spotify/scio/pull/1510>)(#1565 <https://github.com/spotify/scio/pull/1565>)(#1432 <https://github.com/spotify/scio/pull/1432>)(#1614 <https://github.com/spotify/scio/pull/1614>) Add elasticsearch 6 (#1572 <https://github.com/spotify/scio/pull/1572>) Improve AvroType.toSchema annotation error if a case class is not provided (#1609 <https://github.com/spotify/scio/pull/1609>) New scio website <https://spotify.github.com/scio> (#1610 <https://github.com/spotify/scio/pull/1610>) Bug fixes & Improvements Make PTransform names unique #1355 <https://github.com/spotify/scio/issues/1355> #1387 <https://github.com/spotify/scio/pull/1387> Fail for unknown args in ContextAndArgs.typed[T] (#1413 <https://github.com/spotify/scio/pull/1413>) Fix verifyNondeterministic exception in coders (#1418 <https://github.com/spotify/scio/pull/1418>) Fix BigQueryType on refined types (#1424 <https://github.com/spotify/scio/pull/1424>) Fix mergeAccumulators crash (#1428 <https://github.com/spotify/scio/pull/1428>) Set timestamp attribute in JobTest for PubSubIO (#1417 <https://github.com/spotify/scio/pull/1417>) Rework Coder's implicit not found message (again) (#1469 <https://github.com/spotify/scio/pull/1469>) Fix KryoRegistrar scope widenning (#1462 <https://github.com/spotify/scio/pull/1462>) Make compression options in ExtractOps typed (#1449 <https://github.com/spotify/scio/issues/1449>) (#1457 <https://github.com/spotify/scio/pull/1457>) Add back BigQuery schema caching, regression of #1439 <https://github.com/spotify/scio/pull/1439> (#1458 <https://github.com/spotify/scio/pull/1458>) Register default file systems in Scio test context (fix #1455 <https://github.com/spotify/scio/issues/1455>) (#1463 <https://github.com/spotify/scio/pull/1463>) Use coherent defaults accross IO (#1478 <https://github.com/spotify/scio/pull/1478>) Fix scio-repl to use refactored BigQuery client (#1459 <https://github.com/spotify/scio/pull/1459>) Typed argument parsing is broken when name contains camelCase. (#1460 <https://github.com/spotify/scio/pull/1460>) Pubusb topic name was not being set for Messages (#1568 <https://github.com/spotify/scio/pull/1568>) Fix macro generated class directory (#1558 <https://github.com/spotify/scio/pull/1558>) Fix stack overflow when maxByKey is used with explicit ordering (#1560 <https://github.com/spotify/scio/pull/1560>) Fix id and timestamp attributes not being passed in saveAsPubsub (#1559 <https://github.com/spotify/scio/pull/1559>) Fix flatten type inference changing the coder context bound to an implicit parameter(#1551 <https://github.com/spotify/scio/issues/1551>) Fix: use CodeMaterializer in SideOutputCollections (#1548 <https://github.com/spotify/scio/pull/1548>) Default to disabled warning on coders (#1588 <https://github.com/spotify/scio/pull/1588>) Use alternative to deprecated write method (#1592 <https://github.com/spotify/scio/pull/1592>) Simplify BigQueryType query method arg type parsing (#1585 <https://github.com/spotify/scio/pull/1585>) Add rules for TextIO, AvroIO, PubsubIO and BigQueryIO (#1577 <https://github.com/spotify/scio/pull/1577>) #1587 <https://github.com/spotify/scio/issues/1587>: Fix sideoutput potentialy missing coder (#1598 <https://github.com/spotify/scio/pull/1598>) Add region to DataflowResult (#1479 <https://github.com/spotify/scio/pull/1479>) Remove unused autovalue dependency (#1575 <https://github.com/spotify/scio/pull/1575>)
