Pin owner for out-of-tree usermod, and inter-usermod method calls

[blazoncek]

Is your feature request related to a problem? Please describe.
Currently if one needs GPIO for usermod a new entry is added to pin_manager.h to add a new PinOwner. The same goes for usermod ID constant defined in const.h.

If one wants to create out-of-tree (i.e. not included in WLED source code, privately maintained) usermod that requires GPIO allocation there is no way to extend PinOwner enum (except in local/private repository by modifying pin_manager.h) and that defeats usability of such usermod.

The same is true for usermod ID constant defined in const.h.

Describe the solution you'd like
Add extensibility to PinManager to allow addition of custom PinOwner. Remove the need for usermod ID constants or replace it with something more versatile.

Describe alternatives you've considered
Reuse one of the existing PinOwner codes but that is not a safe approach as there might be a clash. Or, provide a patch for upstream to add new owner, but that is no better than the old approach with in-tree usermods.

Additional context
Since usermods were modified to behave like libraries (i.e. no longer included in usermod_list.cpp) it is expected that WLED API would allow extensibility without modifying WLED's source code.

[DedeHai]

that's a good point. IMHO the UM numbering should be taken care of by the usermod manager and dynamically assign the ID's. AFAIK the usermod ID's are not used in any communication protocols and is purely internal. If that is feasible, usermods can just use their ID as a pin owner without conflict.

[blazoncek]

usermod ID's are not used in any communication protocols

That is not entirely true. 4LD, Rotary encoder and Autosave usermods use IDs to identify if usermods are accessible (they were all written by the same author). Then they communicate with each other directly.

[DedeHai]

maybe that can be resolved by a function to get the ID through the usermod name? see ongoing work in #5234

[blazoncek]

That will require usermod to store other usermod's name, which is IMO unfortunate waste of memory.
There is also a necessity to know other usermod's methods (as seen in 4LD, Rotary encoder and Autosave).
API should be enhaced to support that as well.

[softhack007]

"Direct calling of other usermod methods" is a very special case, I haven't seen that in other usermods than the three created by @blazoncek.

That will require usermod to store other usermod's name, which is IMO unfortunate waste of memory.
There is also a necessity to know other usermod's methods (as seen in 4LD, Rotary encoder and Autosave).
API should be enhaced to support that as well.

I'm not sure this would be possible technically. Out-of-Tree usermods are seperate .cpp files, i.e. compiled seperately. If the rotary usermod instance does not have the definition of the display usermod instance, there is no way to call a method that is not part of the base class.

I could imagine that a hack with "weak references" could be possible. This is a trick where the linker tries to resolve a name. If the name exists in the overall binary, you'll get a valid address pointer. If it does not exist, you'll get nullptr. But this hack still requires that rotary and autosave usermods know the methods added in the display usermod instance.

IMHO the UM numbering should be taken care of by the usermod manager and dynamically assign the ID's. AFAIK the usermod ID's are not used in any communication protocols and is purely internal. If that is feasible, usermods can just use their ID as a pin owner without conflict.

@DedeHai I agree

[blazoncek]

@softhack007 before you "accuse" me of creating offending usermods, check commit history. Please.

[blazoncek]

They were all created using valid and public API available at the time of their creation.

[softhack007]

@softhack007 before you "accuse" me of creating offending usermods, check commit history. Please.

Sorry @blazoncek I did not want to "accuse" you. I was thinking these three were your creation, maybe i'm wrong and you just maintained them for some time.

I intended to say that these three usermods are a special case, and they make use of a "side-effect" that in the past all usermods were joined into one "compilation unit" (included by usermod_list.cpp), and the definition of all methods were visible to other usermods. This does not work any more if each usermod has its own .cpp file, unless one usermods can include another usermods ".h" file.

[softhack007]

The pattern used by the autosave usermod to utilize a method in 4LD usermod is like this:

• usermod_v2_four_line_display_ALT has methods that are not part of the base class, but they are public

WLED/usermods/usermod_v2_four_line_display_ALT/usermod_v2_four_line_display_ALT.cpp

Line 563 in d1d9dec

bool FourLineDisplayUsermod::wakeDisplay() {

WLED/usermods/usermod_v2_four_line_display_ALT/usermod_v2_four_line_display_ALT.cpp

Line 586 in d1d9dec

void FourLineDisplayUsermod::overlay(const char* line1, long showHowLong, byte glyphType) {

• usermod_v2_auto_save uses these methods to display some text on the device:

WLED/usermods/usermod_v2_auto_save/usermod_v2_auto_save.cpp

Lines 61 to 63 in d1d9dec

	#ifdef USERMOD_FOUR_LINE_DISPLAY
	FourLineDisplayUsermod* display;
	#endif

WLED/usermods/usermod_v2_auto_save/usermod_v2_auto_save.cpp

Lines 101 to 105 in d1d9dec

	#ifdef USERMOD_FOUR_LINE_DISPLAY
	// This Usermod has enhanced functionality if
	// FourLineDisplayUsermod is available.
	display = (FourLineDisplayUsermod*) UsermodManager::lookup(USERMOD_ID_FOUR_LINE_DISP);
	#endif

WLED/usermods/usermod_v2_auto_save/usermod_v2_auto_save.cpp

Lines 83 to 90 in d1d9dec

	void inline displayOverlay() {
	#ifdef USERMOD_FOUR_LINE_DISPLAY
	if (display != nullptr) {
	display->wakeDisplay();
	display->overlay("Settings", "Auto Saved", 1500);
	}
	#endif
	}

Edit: I just saw that usermod_v2_four_line_display_ALT already has a separate header file, so the solution for autosave could be:

#ifdef USERMOD_FOUR_LINE_DISPLAY
  #include "../usermod_v2_four_line_display_ALT/usermod_v2_four_line_display.h"
#endif

And indeed, we would need to think about a replacement for UsermodManager::lookup(USERMOD_ID_FOUR_LINE_DISP); in case that these usermods were out-of-tree without a fixed ID.

[blazoncek]

There is also PWM_fan usermod (that I wrote) that uses Temperature usermod in a similar way.
Its usefulness is very limited without Temperature usermod.

[DedeHai]

That will require usermod to store other usermod's name, which is IMO unfortunate waste of memory.

correct, it's either "look up by name" (name is fixed) or "look up by ID" (which requires the global list) can't have it both ways. Since this discussion is about getting rid of said list, I do not see any other way than look up by name.

So I think it should go something like this:

• the usermod manager enumerates usermods and gives each compile usermod an ID to be used for pin allocations or UM-to-UM communication (did not look at how that works)
• the usermod manager provides a function to "get the ID" of a usermod from a usermod name
• the usermod manager also provides a function to "get name from ID" to easily build list of UM names for the UI
• all communication through JSON should not be by ID but by usermod name (I think that's already mostly the case)

[willmmiles]

Re pin manager: in an ideal world, the pin manager could store a pointer to the usermod (or its name). If we cannot afford to spend the extra 3 bytes per pin, the next best solution is to store the index in to the usermods table -- this is currently not exposed by um_manager.cpp but would be straightforward to implement. All we'd need are two functions:

int8_t UsermodManager::getIndexOf(Usermod* m) {
   Usermod** m_entry = std::find(_usermod_table_begin, _usermod_table_end, m);
   if (m_entry== _usermod_table_end) return -1;
   return m_entry - _usermod_table_begin;
}

UsermodManager::getByIndex(int8_t index) {
 if ((index >=0) && (index < getCount())) {
   return _usermod_table_begin[index];
 } else {
   return -1;
 }
}

Re inter-usermod communications:
The correct answer for modules that provide data or services that can be used by other modules is to expose an API via a header file that can be included by other modules. That header could expose an extern reference to their usermod object, or any other formal API, such as some accessor functions. There's no need for dynamic lookup at all; whether some other module is included, and where it's placed in memory, are both knowable at compile time. It's perfectly valid for usermods to include headers provided by other usermods, out of tree or otherwise, just like any other library.

For modules that want to optionally enable interactions, eg. "if this usermod is included, pull data from it", this can be done via platformio scripting. The PWM_fan usermod demonstrates how to do this.