#112 Mastodon attached images not saved/shown in GS 2.0-dev

Hi!

One of federation problems I've seen it's that images from Mastodon are not shown at timelines.

Looking between logs and code seems that images and other media came as an [attachment] array that is not managed by GS and not saved with the notice.

Runnig nightly/2.0-dev with Apache and .htaccess sample.

Did increased debug traces so I can document a little more.

When an "object" with a notice from Mastodon came to create_notice() method it includes the array with the attachment.

2020-02-17 13:50:26 LOG_INFO: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] HTTPClient: HTTP GET https://mastodon.social/users/CarmePont/statuses/103674488902743883 - 200 OK
2020-02-17 13:50:26 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [notice] object
Array
(
    [@context] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams
            [1] => Array
                (
                    [ostatus] => http://ostatus.org#
                    [atomUri] => ostatus:atomUri
                    [inReplyToAtomUri] => ostatus:inReplyToAtomUri
                    [conversation] => ostatus:conversation
                    [sensitive] => as:sensitive
                    [toot] => http://joinmastodon.org/ns#
                    [votersCount] => toot:votersCount
                    [blurhash] => toot:blurhash
                    [focalPoint] => Array
                        (
                            [@container] => @list
                            [@id] => toot:focalPoint
                        )

                )

        )

    [id] => https://mastodon.social/users/CarmePont/statuses/103674488902743883
    [type] => Note
    [summary] => 
    [inReplyTo] => 
    [published] => 2020-02-17T13:45:57Z
    [url] => https://mastodon.social/@CarmePont/103674488902743883
    [attributedTo] => https://mastodon.social/users/CarmePont
    [to] => Array
        (
            [0] => https://www.w3.org/ns/activitystreams#Public
        )

    [cc] => Array
        (
            [0] => https://mastodon.social/users/CarmePont/followers
        )

    [sensitive] => 
    [atomUri] => https://mastodon.social/users/CarmePont/statuses/103674488902743883
    [inReplyToAtomUri] => 
    [conversation] => tag:mastodon.social,2020-02-17:objectId=154360488:objectType=Conversation
    [content] => <p>Descerebrat</p>
    [contentMap] => Array
        (
            [ca] => <p>Descerebrat</p>
        )

    [attachment] => Array
        (
            [0] => Array
                (
                    [type] => Document
                    [mediaType] => image/jpeg
                    [url] => https://files.mastodon.social/media_attachments/files/025/269/297/original/1cd32f26e31bf634.jpeg
                    [name] => 
                    [blurhash] => UOHx+-9G_2Rk9G00Rk%M^,ofxut7V@IBofIo
                )

        )

    [tag] => Array
        (
        )

    [replies] => Array
        (
            [id] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies
            [type] => Collection
            [first] => Array
                (
                    [type] => CollectionPage
                    [next] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies?only_other_accounts=true&page=true
                    [partOf] => https://mastodon.social/users/CarmePont/statuses/103674488902743883/replies
                    [items] => Array
                        (
                        )

                )

        )

)

And it's supposed that in that function, attachment should be collected and added to the notice, but it doesn't.

This is the notice content at the end of create_notice().

2020-02-17 13:50:26 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [notice] note
Notice Object
(
    [__table] => notice
    [id] => 16071
    [profile_id] => 556
    [uri] => https://mastodon.social/users/CarmePont/statuses/103674488902743883
    [content] => Descerebrat
    [rendered] => <p>Descerebrat</p>
    [url] => https://mastodon.social/@CarmePont/103674488902743883
    [created] => 2020-02-17 13:50:26
    [modified] => 
    [reply_to] => 
    [is_local] => 0
    [source] => ActivityPub
    [conversation] => 7439
    [repeat_of] => 
    [verb] => http://activitystrea.ms/schema/1.0/post
    [object_type] => http://activitystrea.ms/schema/1.0/note
    [scope] => 0
    [_profile:protected] => Array
        (
            [556] => Profile Object
                (
                    [__table] => profile
                    [id] => 556
                    [nickname] => CarmePont
                    [fullname] => Carme Pont Bonsfills
                    [profileurl] => https://mastodon.social/users/CarmePont
                    [homepage] => 
                    [bio] => 
                    [location] => 
                    [lat] => 
                    [lon] => 
                    [location_id] => 
                    [location_ns] => 
                    [created] => 2020-01-25 11:20:17
                    [modified] => 2020-01-25 11:20:17
                    [_user:protected] => Array
                        (
                        )

                    [_group:protected] => Array
                        (
                        )

                    [_DB_DataObject_version] => 1.11.3
                    [N] => 1
                    [_database_dsn] => 
                    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
                    [_database] => vps13rr4_gsrsn
                    [_query] => Array
                        (
                            [condition] => 
                            [group_by] => 
                            [order_by] => 
                            [having] => 
                            [useindex] => 
                            [limit_start] => 
                            [limit_count] => 
                            [data_select] => *
                            [unions] => Array
                                (
                                )

                            [derive_table] => 
                            [derive_select] => 
                        )

                    [_DB_resultid] => 
                    [_resultFields] => 
                    [_link_loaded] => 
                    [_join] => 
                    [_lastError] => 
                )

        )

    [_attachments:protected] => Array
        (
        )

    [_attentionids:protected] => Array
        (
        )

    [_replies:protected] => Array
        (
        )

    [_groups:protected] => Array
        (
        )

    [_DB_DataObject_version] => 1.11.3
    [N] => 0
    [_database_dsn] => 
    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
    [_database] => vps13rr4_gsrsn
    [_query] => Array
        (
            [condition] => 
            [group_by] => 
            [order_by] => 
            [having] => 
            [useindex] => 
            [limit_start] => 
            [limit_count] => 
            [data_select] => *
            [unions] => Array
                (
                )

            [derive_table] => 
            [derive_select] => 
        )

    [_DB_resultid] => 22
    [_resultFields] => 
    [_link_loaded] => 
    [_join] => 
    [_lastError] => 
)

And at Notice::saveNew() in Notice.php.

2020-02-17 13:50:27 LOG_DEBUG: [gnusocial.sierranorte.red:30873.d5d13ace POST /inbox.json] [colegota] [Notice.php] saveNew() notice
Notice Object
(
    [__table] => notice
    [id] => 16072
    [profile_id] => 205
    [uri] => tag:gnusocial.sierranorte.red,2020-02-17:noticeId=16072:objectType=note
    [content] => RT @CarmePont Descerebrat
    [rendered] => RT @CarmePont Descerebrat
    [url] => 
    [created] => 2020-02-17 13:50:27
    [modified] => 
    [reply_to] => 
    [is_local] => 1
    [source] => ActivityPub
    [conversation] => 7439
    [repeat_of] => 16071
    [verb] => http://activitystrea.ms/schema/1.0/share
    [object_type] => http://activitystrea.ms/schema/1.0/note
    [scope] => 0
    [_profile:protected] => Array
        (
            [205] => Profile Object
                (
                    [__table] => profile
                    [id] => 205
                    [nickname] => Edgecontrol
                    [fullname] => 
                    [profileurl] => https://mastodon.social/users/Edgecontrol
                    [homepage] => 
                    [bio] => 
                    [location] => 
                    [lat] => 
                    [lon] => 
                    [location_id] => 
                    [location_ns] => 
                    [created] => 2020-01-22 14:48:04
                    [modified] => 2020-02-17 13:50:26
                    [_user:protected] => Array
                        (
                        )

                    [_group:protected] => Array
                        (
                        )

                    [_DB_DataObject_version] => 1.11.3
                    [N] => 1
                    [_database_dsn] => 
                    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
                    [_database] => vps13rr4_gsrsn
                    [_query] => Array
                        (
                            [condition] => 
                            [group_by] => 
                            [order_by] => 
                            [having] => 
                            [useindex] => 
                            [limit_start] => 
                            [limit_count] => 
                            [data_select] => *
                            [unions] => Array
                                (
                                )

                            [derive_table] => 
                            [derive_select] => 
                        )

                    [_DB_resultid] => 
                    [_resultFields] => 
                    [_link_loaded] => 
                    [_join] => 
                    [_lastError] => 
                    [avatar] => https://files.mastodon.social/accounts/avatars/001/014/154/original/4245b5e5de31937f.jpeg
                )

        )

    [_attachments:protected] => Array
        (
        )

    [_attentionids:protected] => Array
        (
        )

    [_replies:protected] => Array
        (
        )

    [_groups:protected] => Array
        (
        )

    [_DB_DataObject_version] => 1.11.3
    [N] => 0
    [_database_dsn] => 
    [_database_dsn_md5] => ad6484c22b9d8245b82eccf98b76f560
    [_database] => vps13rr4_gsrsn
    [_query] => Array
        (
            [condition] => 
            [group_by] => 
            [order_by] => 
            [having] => 
            [useindex] => 
            [limit_start] => 
            [limit_count] => 
            [data_select] => *
            [unions] => Array
                (
                )

            [derive_table] => 
            [derive_select] => 
        )

    [_DB_resultid] => 
    [_resultFields] => 
    [_link_loaded] => 
    [_join] => 
    [_lastError] => 
)

Refs:

Original notice at mastodon.social https://mastodon.social/@CarmePont/103674488902743883

Notice at my node https://gnusocial.sierranorte.red/conversation/7439#notice-16071

I think that part is not being handled in plugin(s).

@roytam is correct.

It's true that we don't check the attachments array of the remote notes in ActivityPub (although our notes are sent with one).

We only retrieve a remote attachment if that comes as part of the note's content (in the form of a link) (due to Embed plugin).

We intend to support fetching attachments from remote notices' attachment array in the future, we just didn't implement yet because notes usually come with links to the remote attachments and we didn't want to deal with the disambiguation of what is in the array as well as in the note's body and what's not.

Ideally these fallback urls wouldn't be included in the notice's body when federated via ActivityPub. This is yet something that requires further study and that we've been postponing as we don't see it as a priority for now...

(@roytam who are you btw?)

we just didn't implement yet because notes usually come with links to the remote attachments

IIRC only pleroma does this. mastodon never put attachment link into notice body.

(@roytam who are you btw?)

just a normal user. ;)

We intend to support fetching attachments from remote notices' attachment array in the future

by using StoreRemoteMedia plugin?

yes

but I think we really need a workaround(for example, appending link(s) into notice body) for showing attachment(s) from mastodon AP status since mastodon doesn't put attachment link(s) into notice body, or mastodon AP notice's attachment(s) will be lost.

@roytam: hm.... noted, that's a good idea, we will see the best way having that in mind :)