RFC: Procedural macros in same package as app

[ora-0]

See discussion on internals here.

I feel the need to clarify that this is not about crates, but packages. I had them confused the initial time I wrote the document.

Rendered

[kennytm]

Without changing rustc itself or inserting some hidden prelude I don't see how can the proc-macro be available from crate::proc_macro::*.

Making it available from crate_macros::* is easy though, just add the flag --extern crate_macros=/path/to/target/debug/deps/libxxxxx.dylib.

[ogoffart]

I think having proc macro in the same package as an library crate would be a great thing. (I actually commented that already some years ago: https://internals.rust-lang.org/t/proc-macro-in-an-existing-library-crate/12345/11 )

Maybe the RFC should also specify a few more things such as in what order it should be compiled compared to the build.rs.
It could either be compiled:

• in parallel to the build.rs, to optimize for compile time.
• or before the build.rs, so that build.rs could use the macros.
• or after running the build.rs script, so that metadata and things generated in OUT_DIR would be available. (my preference)

On should also specify what env variable are being set when the proc-macro is being build, and what cfg are being passed.

[ogoffart]

I think build.rs is a much better example that the tests directory, because like proc macros, it needs to be build before the main crate.

[ogoffart]

IMHO, a greater motivation is to avoid having to make several upload to crates.io and making sure to keep them in sync.

[epage]

For me, its less of an issue publishing multiple crates but getting the pattern right. In general, there is a conflict in people wanting to treat a package as a whole workspace the the format complexity that doing so entails (details). I'm generally in favor of encouraging splitting packages.

proc-macros are odd because they have a logical dependency on the library that re-exports them because they generate code that calls into that library. Its easy to overlook this problem and there isn't a great way to declare this relationship today. The options are

• Have the re-exporting package use a = dependency on the proc-macro
• Have the proc-macro have a [target."cfg(false)".dependencies] on a sibling package that the re-exporting package also re-exports

If this also helps towards $crate, then great!

[ogoffart]

Personally, I am not convinced that it should be a directory. I think it could be a single proc-macro.rs file that contains all the proc macro.
The adventage of having a directory with multiple file is that they can then be compiled in parallel, but i am not sure this is a common use-case.

Also, while having convention is great, I think it should be possible to reference explicitly the files in Cargo.toml, just like we have build = "build.rs"

[epage]

I suspect enough proc-macros are bigger than what would be appropriate for a single file

We should also consider a full [proc-macro] table for this so that it can use required-features

[epage]

There would also need to be a way to turn off auto-discovery like package.build = false, package.autobins = false, package.autolib = false, etc

[tgross35]

We could also check for either proc-macro.rs or proc-macro, since most can span a few files but there are some tiny macros that could fit in one. Though I'm not sure it's worth the complexity/confusion to support both, a single extra directory nesting for anything that fits into a single file doesn't seem that bad.

[epage]

We allow that mixing of file or directory with bins, examples, and tests, so there is precedence for it. Just nothing at the top-level atm.

[ora-0]

We should also consider a full [proc-macro] table for this so that it can use required-features

I'm not sure how this would work, since it would depend on the main code being built first, which it can't.

Edit: Ah, I see #3826 (comment)

[ogoffart]

proc_macro is a normal identifier, and there could, in theory, be a module with that name. how about using the macro keyword?

use macro::my_file::my_macro;

[tgross35]

The interactions with rustc need to be discussed here too. Proc macros are extern crates so Cargo could pass --extern=macros=some_crate_built_macros and use macros::some_macro will "just work" without rustc changes. IMO this is the best solution for simplicity.

Anything more magic then that needs rustc changes to somehow reepxort extern macros in a module (proposed in the RFC) or behind the magic macro keyword (above suggestion). This grows the complexity significantly, and I don't think it's worth involving rustc changes for something that could be Cargo-only.

@ora-0 these exact details need to get discussed somewhere, i.e. "how does one reproduce this behavior with two separate crates and the rustc CLI, no Cargo". Tradeoffs should also be discussed.

[tgross35]

(just seeing this is effectively @kennytm's comment at #3826 (comment))

[Noratrieb]

I agree that this should avoid all rustc/language changes and be Cargo-only.

[programmerjake]

I think having the extern crate be named macro or some similar not-previously-allowed name would finally allow $crate and def-site spans to refer to the corresponding library crate rather than the proc-macro itself, which I think is useful enough that it's worth changing rustc.

[workingjubilee]

I don't think minor rustc alterations should be out of the question but obviously I would want a weigh-in from @petrochenkov on any proposals involving preludes or special resolution and I imagine he will prefer a strong justification.

[ogoffart]

[dev-dependencies] is not an option, because Cargo doesn't download these for normal build and wouldn't build them.

It would have to be in [build-dependency], because Cargo would then download and build them for the host.

Another option would be to use [proc-macro-dependencies] which has the advantage of being explicit, but then it adds a new section in Cargo.toml which adds complexity.

[epage]

Another option would be to use [proc-macro-dependencies] which has the advantage of being explicit, but then it adds a new section in Cargo.toml which adds complexity.

Complexity isn't the main concern but it has a major impact on the ecosystem that would need to be worth the cost for all of the relevant tools to be updated to process a new dependency table

[epage]

@ogoffart from #3826 (review)

Maybe the RFC should also specify a few more things such as in what order it should be compiled compared to the build.rs. It could either be compiled:

• in parallel to the build.rs, to optimize for compile time.

• or before the build.rs, so that build.rs could use the macros.

• or after running the build.rs script, so that metadata and things generated in OUT_DIR would be available. (my preference)

(split in a thread to make it not lost)

[SkiFire13]

It should also be specified whether it should be built before or in parallel with the main crate, i.e. if the proc macro should be available to the main crate.

In some cases (e.g. derive macros) the proc macro may not be needed in the main crate, but only by downstream crates. Building the proc macro in parallel with the main crate would thus speed up compilation. See for example this serde issue.

[programmerjake]

For some code, the main crate depends on the proc-macro so it can both export it and use it internally, e.g.: fayalite
So, I think that the proc-macro should be built before the main crate (though I'd be fine if that's configurable).

[ora-0]

It has to be built before the main crate because it is built before lib.

[SkiFire13]

For some code, the main crate depends on the proc-macro so it can both export it and use it internally, e.g.: fayalite

IMO it should be configurable then. Having this dependency by default when not necessary and reducing pipelining (and hence increasing compile times) would be unfortunate.

It has to be built before the main crate because it is built before lib.

What are "main crate" and "lib" for you? They are the same for me. And even if they were different, why does the proc macro need to be built before the lib?

[epage]

@ogoffart from #3826 (review)

On should also specify what env variable are being set when the proc-macro is being build, and what cfg are being passed.

(split in a thread to make it not lost)

[epage]

Like @ogoffart pointed out, this isn't new. It would be good to step through past discussions and compare to where those landed

[epage]

Is this for metaprogramming with reflection?

In that case, another alternative is the work to make declarative macros cover more use cases of proc-macros.

[tgross35]

Since this RFC is mostly a build system change, the alternatives should also be mostly related to the build system. Everything currently in this section would be on the rustc / lang side.

[epage]

Eh, this is a change trying to solve a problem for users. There are multiple ways of solving that problem, build system or not. In general, I think we need to be thinking cross-team more in how we solve problems.

[epage]

It would be good to step through a couple of well known or exotic proc-macros, showing what this would be like. For example, serde has the need for the proc macro to be gated by the derive feature, so how would that look? I called out the answer in #3826 (comment) but this would help find anything else missing and help make this more clear on how this would work

[epage]

When looking at those proc macros, another area to consider is how they do testing and how that maps into this new design

[epage]

What file inside the directory? Is it proc-macro/lib.rs or something else?

[ora-0]

I chose mod.rs, but lib.rs is an option as well

[tgross35]

How exactly does this work - would proc-macro/macro1.rs and proc-macro/macro2.rs be macros that get compiled separately? Or is there single proc-macro crate with the root at proc-macro/lib.rs?

Personally I think the second option is better so it's less confusing to share code among multiple macros. Whatever the decision, the alternative should be mentioned in "rationale and alternatives".

[ora-0]

I have wrote the first revision. Let me know if I missed anything in the feedbacks

A list of important things that have been changed:

1. Add example of a library that has both normal function and macros
2. Add specifications on the configs in cargo.toml
3. Imported via macros::*
4. Use src/macros.rs instead of a proc-macros directory. See the rationale and alternative section, and the example under the explanations section. I believe this is a better system than the original.
5. Clarify that it is built after build.rs, and before lib.rs.

[Jules-Bertholet]

This is missing a specification of the command-line interface. Can I do cargo build --macros?

[ora-0]

This is missing a specification of the command-line interface. Can I do cargo build --macros?

Yes, I'll add this to the rfc

[programmerjake]

I don't like that src/macros.rs ends up becoming the proc-macro entry point if that file is available, I expect there's quite a few crates that already have a src/macros.rs file since that's where they stick all their non-procedural macros. I think we should instead have the entry point be somewhere where code is currently unlikely to be, so instead of src/macros.rs, I think it should be in macros/lib.rs or similar.

[ora-0]

I don't think macros/lib.rs makes sense. It wouldn't allow for macros.rs to use common.rs, if you look in the example. (also see the rationales and alternatives section on that)

We could rename it to procmacros.rs. There shouldn't be much projects with that. Even if they do they can change the path in cargo.toml.

[ora-0]

I'm not sure about the casing on that one though. Should it have no spaces, be kebab-case or snake_case?

[programmerjake]

I don't think macros/lib.rs makes sense. It wouldn't allow for macros.rs to use common.rs, if you look in the example.

you can import files from src/:

// in macros/lib.rs
#[path = "../src/foo.rs"]
mod foo;

use foo::Foo;

// in src/foo.rs
pub(crate) struct Foo;

in any case, I don't expect proc macros to need to share code with their corresponding libraries all that often.

imo any method (using #[path] or not) of sharing source files between multiple different crates (the lib, bins, proc-macros) shouldn't be encouraged, since they act like totally separate modules that happen to have the same source code, but they don't define the same types, functions, or other items -- you can't use a type from one instance of the module and pass it to another instance. generally they're just confusing to use.

[ora-0]

I guess. I was mainly basing it off of one suggestion made in the pre-rfc. Still not too sure where to put it though. I think procmacros.rs is alright.

Either case the user could always modify it in the cargo.toml

[ora-0]

Sure, I'll put it back as proc-macro/lib.rs in the next revision. But I'm not sure what you mean by

I think the difference there is that main.rs will often use the library while macro source code is usually completely different from library code

Could you clarify please?

[tgross35]

Usually if you have a package with both a main.rs and a lib.rs (plus the rest of the library), the library has basically everything and then main.rs is a tiny single-file wrapper to call library functions. So it's small and shares the logic, seems reasonable enough to keep in the same src/ directory.

Proc macros on the other hand tend to have pretty distinct code from non-proc-macro crates, and they're probably more likely to span a couple of files. I don't think we want to encourage directory structures like this:

src/
    lib.rs         # <- library entrypoint
    math.rs
    parse.rs       # <- library functionality or TokenTree parser?
    proc-macro.rs  # <- proc macro entrypoint
    util.rs
    util_but_for_macros.rs

Where some names clash, or you can't tell at a glance whether files belong to the library or to a proc macro.

[programmerjake]

usually if there's a executable for using some library, you'd put the command-line parsing in main.rs and just call the exported functions from the library crate. by contrast proc-macros don't usually need any functions from their corresponding library (it's often the other way around where the library needs to call the proc-macro).

e.g. if you have some rust project named json_prettify, you can have:
src/main.rs:

// use a function from the library crate, no `mod` needed
use json_prettify::format_json;

fn main() -> std::io::Result<()> {
    // TODO: parse command line
    format_json(std::io::stdin(), std::io::stdout())
}

src/lib.rs:

use serde_json::{from_reader, to_writer_pretty, Value};

pub fn format_json<R: std::io::Read, W: std::io::Write>(input: R, output: W) -> std::io::Result<()> {
    let json: Value = from_reader(input)?;
    to_writer_pretty(output, &json)
}

[epage]

I agree with not sharing src/.

I feel like allowing lib.rs and main.rs to be in the same directory has caused a lot of confusion because users may mod something they actually mean to use from the library. If it wasn't so convenient for bin-only packages, I'd be considering deprecating src/main.rs.

[programmerjake]

started a new topic in internals: Deprecating having both src/lib.rs and src/main.rs in the same package

[epage]

I have wrote the first revision. Let me know if I missed anything in the feedbacks

@ora-0 as a tip for RFCs,

• if you make each edit in its own commit, the commits can describe whats changed
• link to the commit from the relevant threads so people can easily find how you've responded to their feedback

[epage]

Been thinking about this more and I'm unsure. If we tell the proc macro at compilation time, Cargo would need to know a unique identifier for the re-exporting crate. However, I'm not sure how easily we can determine that unique identifier when building a dependent.

If we could somehow bind $crate later in the process, that might work but Cargo won't know what all to apply that binding to.

If we can't do this, how much would this affect this proposal?

[ora-0]

If we can't do this, how much would this affect this proposal?

Not much, I would say. When I was first ideating the proposal $crate was not part of my mind in the first place; it wasn't a big motivating factor, with mainly ux being on my mind. I think it would be better to leave the problem of the implementation to it for later.

[joshtriplett]

It'd already be feasible, today, to make wrapper macro_rules! macros in the main crate, and in those macros, pass $crate to an underlying proc macro.

That's not ideal, and it'd be nice to make it more usable, but that seems like a baseline starting point.

[epage]

Could you go into the motivation for --macros?

This is an ineresting one because build scripts do not have a --build-script but we do have --lib and --bins. What makes this fall into one camp or the other?

[ora-0]

It's because when one types cargo build, the build script is compiled and ran, but the others, including macros, are only compiled.

[epage]

That doesn't quite make sense to me. We could have a cargo build --build-script and just build it and not run it. Build scripts are not an addressable target / artifact in cargo today but a means to an end. Conceptually, the proc macro crate is also a means to an end but it will have a final artifact.

Whether to have the flag or not can also be affected by whether we support unit tests inside of it (we don't for build scripts) so you can run cargo test --macros.

Note: even if nothing is changed from this, this fleshes out the rationale and should be included / summarized in the RFC.

Side note: whats interesting is we're working on moving build scripts out into their own packages while this does the opposite for proc macros. I think the use cases are different enough that one does not affect the other but kind of interesting to observe.

[epage]

This should talk through naming

• --macros flag
• extern crate macros;
• [proc-macro]
• proc-macro/lib.rs

Why aren't we consistent? Why have the name in one direction or the other?

[ora-0]

That's true. Do we prefer proc-macro or macros? The first is more explicit but the second is shorter. I guess the second one could create some confusion on whether or not it's procedural or declarative, but that might not be that big of a problem. What do you think?

[epage]

I would lean towards proc-macro as that is already what we stabilized, see https://doc.rust-lang.org/cargo/reference/cargo-targets.html#configuring-a-target

[ora-0]

ff89040

[epage]

Could you add a discussion under the Rationale and Alternatives that speaks to what was considered for naming and why the current option was chosen?

[SOF3]

Is there any particular impact on rustdoc?

• Should docs of the proc macro crate be rendered on docs.rs
• What is the path of the proc macro crate under target/doc
• What happens if there are multiple packages with a proc macro crate

Unlike build crate, proc macro crates export items directly visible to downstream packages through reexporting, so documentation might be necessary, although it might just be inlined in the doc itself. But should it still be documented with --document-private-items?

[ora-0]

Should docs of the proc macro crate be rendered on docs.rs

Probably not, since we the api won't be publicly available unless reexported.

What is the path of the proc macro crate under target/doc

It would be whatever the name of the crate is. By default it would be "macros", but can be changed in cargo.toml.

What happens if there are multiple packages with a proc macro crate

hmm... Perhaps we should change the default name to be "crate-name-macros" to avoid collisions, or "crate-name-proc-macros" depending or which one we decide to go with. The latter does feel a bit too long for my taste tho.

And regarding the --document-private-items, I believe it should document it. I'll add a section that details the behaviours sometime.

[epage]

Above you say that proc-macro.test = true is the default but this says cargo test --macros is unsupported. This seems contradictory.

[ora-0]

Oh yea, I wrote the test = true thinking it was for integration tests. That's my bad

Edit: I made cargo test --macros not possible as I was thinking that you can't call macros from the same crate, but on further thought it'll still be possible to call them, but passing it from the result of TokenStream::from_str.

[ora-0]

c7f34c5

[epage]

By the link you gave, these are env variables set on the proc-macro build target?

We don't set other CARGO_CFG_* on general build targets, so why CARGO_CFG_PROC_MACRO?

[ora-0]

I swear I remember seeing somewhere that CARGO_CFG_PROC_MACRO exists... I might've confused it with something else I guess.

[ora-0]

3f94b5b

[epage]

Cargo will take "all" cfgs for a library and turn them into CARGO_CFG_* for the library's build script

[ora-0]

Do we have any ideas on what name the new target should be? Currently it's macros but it may cause name collisions with --document-private-items.

One way to solve this is to only display the macros crate of the current package, which isn't a bad option in my opinion, since it shouldn't really be exposed anyways, except through lib. Unfortunately we can't use proc_macro to stay consistent as it's already taken.

Or we could prefix it with the package name, something like "crate_name_macros". But libraries like Tokio are already doing something like this, with the tokio-macros crate, so it may cause collisions that way.

The final option is "crate_name_proc_macro", which would be consistent with what we have, but feels a bit too long for me, though it would solve the issue of the second option.

[programmerjake]

Do we have any ideas on what name the new target should be? Currently it's macros but it may cause name collisions with --document-private-items.

assuming you're talking about paths in the output of rustdoc, you could use some character that's illegal in crate identifiers but legal in file-system paths and URLs, e.g. serde.proc-macro, giving you a URL like
https://docs.rs/serde/latest/serde.proc-macro/derive.Deserialize.html

[hanna-kruppe]

I'm missing a clear statement which proc-macros is this feature for?

The motivation section focuses on the ease of introducing a proc macro in a library that currently doesn't have one. I have sympathy for that, the proc-macro <-> build script/decl-macro switcheroo isn't fun. If the feature was tightly scoped to experimentation and simple library-internal use, the build order questions discussed in #3826 (comment) have a clear, simple answer. Dependents also wouldn't need to know that the proc macro exists, let alone opt in/out with crate features (cc #3826 (comment)).

But it seems quite a few people commenting here see this feature as a general replacement for "library re-exports proc-macros" -- and it's not like we can stop people from re-exporting the proc macros like that, even if that wasn't the intended usage. If the feature isn't designed with that in mind then there's a risk it'll end up as a newbie trap: easier to get started, but once you get serious you have to dig yourself back out of the downsides of the easier way. But in that case, the feature will become much more complicated. Thus, my questions: Is it an explicit design goal that essentially any library providing proc macros could use this feature? If not, what's out of scope and how would people pick between using this feature and making a separate crate?

[epage]

What do you see as the downsides to designing a proc-macro in this way? I don't think I've seen those raised yet in this discussion.

I've seen this as a general way of doing proc-macros and would see this feature not to be worth it (or maybe actively opposed) if this is for smaller scope and then people grow out of it. In general, I'm not thrilled with cramming everything into a single package to avoid having a workspace and feel that to meet what people want in doing that would move cargo away from what its good for. That said, I feel like this doesn't run into those problems and can help avoid some proc-macro problems (e.g. the proc-macro logically depends on the library it re-exports but can't declare that relationship nor is it obvious of a problem for users to solve or how to solve it).

[hanna-kruppe]

FWIW, I share your gut feeling that this feature probably isn't worth it if it's only useful for small, simple macros, but I didn't want to prejudice my comment with that. The downsides I see with trying to do "everything" are mostly the complexity that I expect will be required to address all of the concerns raised in other comment threads. To avoid introducing downsides compared to a separate package, this feature will need:

• Support for both the "proc-macro is compiled independently of library" use case (generally faster builds) and the "library also uses its associated proc-macro internally" use cases. (RFC: Procedural macros in same package as app #3826 (comment))
• Ability to make the proc-macro part of the package optional, like the common derive = ["dep:foo_derive"] feature. In general this needs to be a proper feature because it may interact in complex ways with ordinary features of the package. (RFC: Procedural macros in same package as app #3826 (comment))
• Ability for some crates to depend only on the library without pulling in the proc-macro parts, even if other parts of the dependency graph require both. For example, crates like time and jiff depend only on serde and write impls by hand; as long as other crates in the build depend directly on serde_derive (not via serde/derive) the former crates can still be compiled independently of the proc-macro and its dependencies.

All of these points (and others we may find) can be addressed in theory. But not all of them seem to have straightforward solutions that fit in the scope of what a package currently is. For example, something like required-features on a packages' proc-macros could serve the second bullet point, but the third one needs a way to say "I only depend on this part of that package" (here: the lib, not the proc-macro) which doesn't currently exist and seems like a step towards "N things in 1 package instead of N packages in a workspace".

can help avoid some proc-macro problems (e.g. the proc-macro logically depends on the library it re-exports but can't declare that relationship nor is it obvious of a problem for users to solve or how to solve it).

This is a problem worth solving, but I'd prefer a solution that existing libraries with separate foo_derive packages can adopt without breaking their users who depend directly on foo_derive (most notably, serde). For example, if the proc-macro package could directly express which library package+version it's coupled in its manifest, that only requires a sufficiently new MSRV but no package restructuring.

[epage]

Maybe there is something I'm missing but I feel like the answers to these questions are fairly simple; the author just needs to take the time to enumerate and write it up.

For example:

Support for both the "proc-macro is compiled independently of library" use case (generally faster builds) and the "library also uses its associated proc-macro internally" use cases.

This is no different than the regular proc-macro situation. If you are like serde today, in both cases the core functionality will be blocked on the derive building. However, if you have a derive like clap or serde after serde-rs/serde#2608, the proc-macro and the core library will be built in parallel and the re-export library will be blocked on both. Again, no difference between a separate package and same package.

Ability to make the proc-macro part of the package optional, like the common derive = ["dep:foo_derive"] feature. In general this needs to be a proper feature because it may interact in complex ways with ordinary features of the package

This is why I brought up [proc-macro] table so you can do

[proc-macro]
required-features = ["derive"]

And with that, your proc-macro is now skipped if derive is not enabled using existing Cargo functionality.

You mentioned a problem with this related to addressing one of N libraries in a package. That didn't quite make sense to me but if its related to the other points, then I addressed those.

Ability for some crates to depend only on the library without pulling in the proc-macro parts, even if other parts of the dependency graph require both.

While true that if everyone depended on serde_derive, rather than serde/derive, people could depend on serde without a slow down. This is unlikely to ever happen though so this is a benefit in theory. Instead, we need serde-rs/serde#2608 wrapped up and then libraries like serde_json can depend on serde_core without the need for cooperation from every dependency. As a bonus, this will mean that we can see a serde_derive v2!

This is a problem worth solving, but I'd prefer a solution that existing libraries with separate foo_derive packages can adopt without breaking their users who depend directly on foo_derive (most notably, serde). For example, if the proc-macro package could rust-lang/rust#54363 (comment) in its manifest, that only requires a sufficiently new MSRV but no package restructuring.

That proposal has a lot of complexity to it that I do not see it being likely to happen.

This introduces cycles between packages, particularly for publishing. There are ways to solve problems with that but taking just one part as an example, this would require a new publish API which has been stalled out for years.

This also takes a concern I have with this proposal being able to solve $crate (#3826 (comment)) and greatly increase the complexity for solving it because it would extend from determining information from a strict child-parent relationship to an arbitrary relationship as we work through the build graph.

[hanna-kruppe]

This is no different than the regular proc-macro situation. If you are like serde today, in both cases the core functionality will be blocked on the derive building. However, if you have a derive like clap or serde after serde-rs/serde#2608, the proc-macro and the core library will be built in parallel and the re-export library will be blocked on both. Again, no difference between a separate package and same package.

The difference is that with separate packages, it's clear how any of the three options can expressed. The library either depends on the associated proc-macro package or doesn't; a facade crate like clap is just a separate package that depends on both the library and the proc-macro. If library and proc-macro are part of the same package, there needs to be some new way to express these dependencies (possibly optional/feature-gated) between the library and the proc macros within the same package. It's less clear to me what this would look like than with the required-features thing.

As I've said, I'm sure there's many ways to design support for this. But without having seen a complete proposal for all aspects, I'm no sure whether the end result will truly be better than the status quo.

While true that if everyone depended on serde_derive, rather than serde/derive, people could depend on serde without a slow down. This is unlikely to ever happen though so this is a benefit in theory.

It's not a theoretical benefit. Inertia is strong, of course, but I've seen a few big dependency trees reach this goal. For example, most configurations of wasmtime before the switch from bincode to postcard achieved it, as long as you disabled the profiling feature (and the postcard thing is fixable). Also, new libraries with derives can do the "please depend on foo and foo-derive separately" thing from the start and avoid offering a derive feature in the first place (I've done that for a new library I started recently). This should remain expressible with "proc-macro in same package" if we want all proc macros to adopt this style (modulo MSRV and backwards compatibility issues).

Besides, I don't understand how a clap-style (or serde/serde_core-style) split would interact with this RFC. Assuming the "at most one library per package" rule remains in place, that means there's still going to be at least two packages involved. If that's going to be the recommended approach, the simplicity the RFC cites as motivation is mostly lost. If you put the proc-macro into the package that re-exports the underlying library crate (e.g., clap_derive becomes a [proc-macro] of clap while clap_builder remains separate) you still have the problem that the package with the derive and the package with the traits being derived are separate and their version needs to match exactly. If the $crate-for-proc-macro angle works out (seems very speculative to me at this point) then you'd keep that benefit, but the other benefits seem much less applicable than in the "one package for everything" case.

[ora-0]

Thus, my questions: Is it an explicit design goal that essentially any library providing proc macros could use this feature?

It wasn't an initial goal, but in the pre-RFC and RFC there has been a need to do it so a lot of the revisions do have that in mind. There is still a lot of room of iterations.

Ability to make the proc-macro part of the package optional... Ability for some crates to depend only on the library without pulling in the proc-macro parts, even if other parts of the dependency graph require both.

You can just use conditional compilation on your proc macros. It'll just be like how you gate all other features. Unless I'm overlooking something.

It feels that some of the criticisms, while valid, are a bit too fixated on the idea of the separation between proc-macro packages and library packages. There was really no reason for them to be separated in the first place, except for build system, which I think most of us can agree is an unfortunate blocker. In an ideal world, proc macros and code can coexist together, and this is a close equivalent.

[epage]

Besides, I don't understand how a clap-style (or serde/serde_core-style) split would interact with this RFC. Assuming the "at most one library per package" rule remains in place, that means there's still going to be at least two packages involved. If that's going to be the recommended approach, the simplicity the RFC cites as motivation is mostly lost.

I fundamentally disagree with the motivation of the RFC. If that motivation was the sole motivation, I would be a hard "no" on this. imo if people have a problem with managing a workspace, we should work to improve workspaces. I mentioned earlier in this thread (#3826 (comment)) that I generally disagree with cramming everything into a package. For more on this topic, see https://blog.rust-lang.org/inside-rust/2024/02/13/this-development-cycle-in-cargo-1-77/#when-to-use-packages-or-workspaces

As I mentioned before (#3826 (comment)), the main benefit this gives us is a happy path for declaring the relationship between the proc-macro and the library it generates code against. As a secondary motivation is I see this as a fundamental step towards unblocking $crate though there are likely other blockers.

So for clap, I would merge clap_derive into clap and keep clap_builder as a separate package.

The difference is that with separate packages, it's clear how any of the three options can expressed. The library either depends on the associated proc-macro package or doesn't; a facade crate like clap is just a separate package that depends on both the library and the proc-macro. If library and proc-macro are part of the same package, there needs to be some new way to express these dependencies (possibly optional/feature-gated) between the library and the proc macros within the same package. It's less clear to me what this would look like than with the required-features thing.

As I've said, I'm sure there's many ways to design support for this. But without having seen a complete proposal for all aspects, I'm no sure whether the end result will truly be better than the status quo.

I think there is something missing in our communication because you seem to still feel there are significant open questions on this point while I feel that I explain how this naturally falls out of the design and there isn't anything special to this (though the RFC does need to explicitly call this all out).

If there are lingering concerns, feel free to reach out to me for a call for us to work this out.

It's not a theoretical benefit. Inertia is strong, of course, but I've seen a few big dependency trees reach this goal. For example, most configurations of wasmtime before the switch from bincode to postcard achieved it, as long as you disabled the profiling feature (and jamesmunns/postcard#130). Also, new libraries with derives can do the "please depend on foo and foo-derive separately" thing from the start and avoid offering a derive feature in the first place (I've done that for a new library I started recently). This should remain expressible with "proc-macro in same package" if we want all proc macros to adopt this style (modulo MSRV and backwards compatibility issues).

Ok, so I take back my saying its theoretical. I still think it is not a workflow we should prioritize in our designs. Cargo is an opinionated tool focused on usability. Having to do that much precise coordination across a dependency tree seems like the wrong answer with Cargo's design.

[hanna-kruppe]

Thank you for elaborating on your perspective on what the RFC should be, @epage, this helps me understand your previous comments better. Also thanks for offering a call, but I think at this point it's probably a better use of everyone's time to let the RFC catch up with the discussion and propose a more detailed design. Then I'll open more focused comment threads if I still have concerns, because then I'll be able to phrase them more precisely since I'll have less guessing to do about what precisely is proposed.

The one thing that I'd like to ask you which the updated RFC probably won't answer is about your plan to merge clap_derive into clap. I'm not sure whether you think the main benefit of the RFC (happy path for declaring relationship between proc-macro and the library is generated code for) applies to that case. I don't think it does, because the derive would still be in a separate package (clap) from the traits that the derive implements (clap_builder), i.e., you'd still need clap to depend on a specific, pinned version of clap_builder. Am I missing something or are you not expecting this benefit to apply in this setup? (Of course, proc-macro $crate, if it works out, would still be useful in this case.)

@ora-0 I'd say I care more about the build system POV (which units of work exist in the build and how they depend on each other) than about packages per se. In essence, if the goal is that every proc macro should share a package with its associated library, then I'm asking what that means for the dependency graph at the build system level. If some useful dependency graphs would be inexpressible in that future style, then I'd consider that a downside of the RFC and would like to discuss the impact. Conversely, if the RFC doesn't restrict our ability to massage the work done by the build system in the most favorable shape (compared to the status quo), I'd like the RFC to spell out how this is possible.

[epage]

The one thing that I'd like to ask you which the updated RFC probably won't answer is about your plan to merge clap_derive into clap. I'm not sure whether you think the main benefit of the RFC (happy path for declaring relationship between proc-macro and the library is generated code for) applies to that case. I don't think it does, because the derive would still be in a separate package (clap) from the traits that the derive implements (clap_builder), i.e., you'd still need clap to depend on a specific, pinned version of clap_builder. Am I missing something or are you not expecting this benefit to apply in this setup? (Of course, proc-macro $crate, if it works out, would still be useful in this case.)

There are two common dependency problems when you have a lib -> lib_derive relationship

• lib requires a minimum version of lib_derive but lib_derive needs to declare a minimum version on lib.
  • The simplest way of solving that is with lib using lib_derive = "=1.0.100".
  • If you merge lib_derive into the lib package, this problem goes away.
  • If you also have a lib_core, that doesn't matter because lib_derive only needed a minimum version requirement and lib already has that on lib_core
• lib_derive using internal APIs of lib which requires a = requirement.
  • If you merge them, this goes away.
  • If lib re-exports a lib_core which actually has the internal API, then you will still need a = requirement.
  • Hopefully instead lib will only re-export public APIs from lib_core and the private API used by the former lib_derive could live inside of lib.

Since clap_derive does not generate code using a private API from clap, it could be merged into clap without a problem. In fact, I could remove the use of = in clap today by having clap_derive declare a target.cfg(false).dependencies on clap_builder. I don't remember why I haven't done that.

serde_derive does use private APIs. I've not followed the serde_core PR too closely on whether those are living inside of serde or serde_core.