Hi all,
While upgrading Avro from 1.12.0 to 1.12.2 (PR #24992, still in
review) I ran into a behaviour change that I think everyone working on
schemas, Functions or connectors should know about, because it also
affects user applications on upgrade and not just our own code.
Short version: as of Avro 1.12.2, Avro refuses to reflect over any
class that is not explicitly trusted. Pulsar's AVRO schemas are built
on Avro's reflection API, so without an allow-list every
Schema.AVRO(MyPojo.class) encode and decode throws SecurityException.
(Schema.JSON is unaffected - it goes through Jackson.)
WHAT CHANGED, AND IN WHICH RELEASE
----------------------------------
It is worth separating the two releases, since the change that causes
this landed in 1.12.2 rather than in 1.12.1.
Avro 1.12.1 (October 2025) shipped 4 security fixes. One of them
removed the default trusted packages, which used to be:
java.lang, java.math, java.io, java.net, org.apache.avro.reflect
But in 1.12.1 the allow-list was still consulted in exactly one place,
SpecificDatumReader.findStringClass. That means it only guarded
classes named by the "java-class" / "java-key-class" schema
properties. Record, enum and fixed types were still resolved with no
check at all. The class org.apache.avro.util.ClassSecurityValidator
does not exist in 1.12.1.
Avro 1.12.2 is where the scope changed. AVRO-4189 introduced
ClassSecurityValidator and moved the check into ClassUtils.forName, so
it now guards *every* reflective class resolution. What remains
trusted by default is 16 hardcoded JDK classes and no packages at all:
Boolean, Byte, Character, CharSequence, Double, Enum, Float, Integer,
Long, Number, Object, Short, String, Void, BigDecimal, BigInteger
That matters because SpecificData.getClass(Schema) resolves a class
for every RECORD, ENUM and FIXED schema, and
ReflectData.populateEncoderCache calls it via getCustomEncoding on
both ReflectDatumWriter.write and ReflectDatumReader.read. (Those
getCustomEncoding call sites are themselves new in 1.12.1, from
AVRO-4165.) So both the produce and the consume path go through the
check now, for every named type in the schema.
This is a significant tightening: in 1.12.1 the allow-list applied
only to classes named by java-class / java-key-class, and in 1.12.2 it
applies to every reflective class resolution. Plan for it as a
deliberate hardening that we adapt to.
Note that staying on 1.12.1 is not a workaround: AVRO-3940 in 1.12.1
introduced a StackOverflowError in ReflectData.getSchema for any POJO
with a recursive field (AVRO-4209), and that is only fixed in 1.12.2.
WHAT ACTUALLY BREAKS
--------------------
If you do nothing, on Avro 1.12.2:
* Producing or consuming with Schema.AVRO(MyPojo.class) fails with
java.lang.SecurityException: Forbidden com.example.MyPojo!
This class is not trusted to be included in Avro schemas.
wrapped in SchemaSerializationException. This hits both
directions, not just produce.
Schema.JSON is NOT affected: it derives an Avro schema for the
SchemaInfo but reads and writes through Jackson, so it never resolves
a class reflectively. Neither is Schema.AUTO_CONSUME over a record
schema, which stays generic. Schema.PROTOBUF is affected, but at
schema-construction time rather than on produce - avro-protobuf
resolves the generated message classes while building the schema.
* The failure is per named type, so it is not enough to trust the
top-level POJO. Every nested record and every enum reachable from it
is resolved separately, including types from other packages.
* A field's *declared* collection type is recorded by Avro as a
"java-class" property on the generated array/map schema and resolved
reflectively too. So a field declared List<Foo> needs java.util.List
trusted, a HashSet field needs java.util.HashSet, and so on.
* Nested classes have to be named with the binary '$' form. Avro
derives the schema name as com.example.Outer.Inner but resolves the
class by retrying with '$' separators, and the validator matches on
Class.getName(). An allow-list entry written with dots silently never
matches.
* @Stringable types are affected as before, but the previous
defaults are gone, so java.net.URI, java.net.URL and java.io.File now
need trusting explicitly. BigDecimal, BigInteger and Integer are still
covered by the hardcoded set.
* One non-obvious case: Schema.AUTO_CONSUME() against a topic whose
schema is Schema.JSON(SomeEnum.class). Nearly every POJO produces a
top-level Avro RECORD, and records are handled generically without
resolving any class. An enum is the exception - it produces a
top-level ENUM, and for a JSON-typed schema AutoConsumeSchema resolves
the class named in the *writer's* schema in order to rebuild a reader
schema for it. So the consumer needs the enum trusted, and the error
names it with the binary '$' form even though the schema spells it
with dots:
java.lang.SecurityException: Forbidden com.example.Outer$Colour!
Bare String or Integer POJOs are not affected - they produce plain
"string" / "int" schemas with nothing to resolve - and an AVRO-typed
enum schema takes a different branch that does not resolve either. It
is a narrow case, but not one anybody would predict from "Avro schemas
need trusted classes".
Two further 1.12.1 changes are worth flagging while we are here:
* AVRO-3230 enabled the fast reader by default. It reports malformed
data as AvroTypeException where the classic path raised
IndexOutOfBoundsException, so code that catches narrow exception types
around an Avro decode can now leak an Avro exception to callers.
* AVRO-4133 makes avro-protobuf emit a "default" attribute inside
generated enum schemas, which changes the schema JSON produced for a
protobuf class.
WHAT PULSAR DOES ABOUT IT (PR #24992)
-------------------------------------
Pulsar cannot know an application's classes, so the PR only takes
responsibility for what Pulsar itself serializes and for the code
Pulsar runs on the deployment's behalf.
* A new public class,
org.apache.pulsar.client.schema.AvroTrustedClasses, owns the trust and
installs it into Avro's global validator. It composes with whatever
validator is already installed, so Avro's own defaults, the
SERIALIZABLE_* system properties and any policy the application
installed itself all keep working. Its signatures use only Class,
String, ClassLoader and java.util.function.Predicate - deliberately no
Avro types, so it behaves the same in the shaded and unshaded clients.
* BrokerAvroTrustedClasses declares the broker's own Avro-serialized
types: transaction buffer snapshots, topic policy events on
__change_events, and the metadata events used by the metadata
synchronizer.
* FunctionAvroTrustedClasses trusts the class loader Pulsar created
for a deployed function or connector, so one using Schema.AVRO over
its own POJOs keeps working with no configuration, on all three
runtimes. Scoping to the class loader is deliberately narrower than a
wildcard: only the classes actually deployed become trusted, not
everything on the class path.
WHAT CLIENT APPLICATIONS HAVE TO DO
-----------------------------------
If your application uses Schema.AVRO with your own POJOs, declare them:
import org.apache.pulsar.client.schema.AvroTrustedClasses;
// Once, before the first producer or consumer.
AvroTrustedClasses.trust(Order.class);
trust(Class...) walks the schema Avro derives, so one call also covers
the nested records and enums it references - including ones in other
packages - and the declared collection and @Stringable types its
fields carry. That last part matters more than it sounds: a POJO with
a List field needs java.util.List trusted too, which is why "just
trust my model package" is usually not enough on its own. There is
also trustExactly(Class...) when you want only what you named,
trustClasses(String...) for classes you cannot reference at compile
time, trustPackages(String...), trustClassLoader(...) for plugin code
you load yourself, and trust(Predicate<Class<?>>) as an escape hatch.
A warning about Avro's own advice, because it is the first thing users
will read. The SecurityException message says to set
org.apache.avro.SERIALIZABLE_CLASSES / SERIALIZABLE_PACKAGES "or set
them via the API (see org.apache.avro.util.ClassSecurityValidator)".
For anyone on the shaded client - pulsar-client or pulsar-client-all,
which is the usual dependency - BOTH of those are wrong:
* The class is not on their class path. Shading relocates it to
org.apache.pulsar.shade.org.apache.avro.util.ClassSecurityValidator,
so the snippet does not compile. Adding org.apache.avro:avro to make
it compile is worse: it compiles, runs, configures a second copy of
the class and the SecurityException fires unchanged.
* The property name is relocated with it, so
-Dorg.apache.avro.SERIALIZABLE_PACKAGES is silently inert there.
So a shaded-client user who follows the exception gets a
byte-identical failure and concludes the upgrade is broken. That is
the main reason the PR adds AvroTrustedClasses rather than pointing at
Avro's API: we cannot change Avro's message, but we can give people
something that works either way.
The system properties do work if you use the right name for the artifact:
# pulsar-client / pulsar-client-all (shaded - the usual dependency)
-Dorg.apache.pulsar.shade.org.apache.avro.SERIALIZABLE_PACKAGES=com.example.model
# pulsar-client-original (unshaded), broker, proxy and other components
-Dorg.apache.avro.SERIALIZABLE_PACKAGES=com.example.model
SERIALIZABLE_CLASSES takes individual class names instead. Both are
read in a static initializer, so they have to be set before anything
in the JVM has touched Avro - a command-line -D always works, and
System.setProperty() works only if it runs first. Note that a package
on its own will not cover the java.util.List in the example above; you
would need SERIALIZABLE_CLASSES for that too, which is exactly the
bookkeeping trust(Class...) does for you.
It is also possible to turn the check off entirely, by trusting every
package - again with a different property name depending on the
artifact:
# pulsar-client / pulsar-client-all (shaded)
-Dorg.apache.pulsar.shade.org.apache.avro.SERIALIZABLE_PACKAGES=*
# pulsar-client-original (unshaded), broker, proxy and other components
-Dorg.apache.avro.SERIALIZABLE_PACKAGES=*
Avro treats "*" as trust-all, so this restores the pre-1.12.2
behaviour. It is there if the upgrade breaks an application and you
need it working again while you work out which classes to trust.
Whether to keep it that way is a policy decision for each deployment
to make on its own assessment.
This needs a release note, and I think it needs a documentation page
too. Feedback on the plan is welcome.
Thanks, Lari
